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(54) METHOD OF PROCESSING, TRANSMITTING AND RECEIVING DYNAMIC IMAGE DATA AND 
APPARATUS THEREFOR 

(57) A reception control section 1 1 for receiving the 
Information including data and its transmission format 
information from a memory or communication channel, 
a separating section 12 for analyzing and separating 
received information, a transmitting section 13 for trans- 
mitting information to a memory or transmission chan- 
nel, a vkieo extending section 14 for extending a video^ 
and video-extensfon control section 15 control the 
processing state of said video extending section 14 for 
extending at least one or more videos and a video syn- 
thesizing apparatus constituted with a video synthesiz- 
ing section 16 for synthesizing videos in accordance 
with extended Information, an output section 17 for out- 
putting a synthesized result, and a terminal control sec- 
tion 18 for controlling the above means makes it 
possible to synthesize a plurality of videos at the same 
time and correspond to a dynamic change of transmis- 
sion format Information. 




Printed by Xerox (UK) Business Services 
2.16.7/3.6 



1 



EP0905 976A1 



2 



Description 

Technical Field 

[0001] The present invention relates to audio-video 
transmitter and audio-video receiver, data-processing 
apparatus and method, waveform-data-transmitting 
methcxJ and apparatus and waveform-data-receiving 
method and apparatus, and video-transmitting method 
and apparatus and video-receiving method and appara- 
tus. 

Background Art 

[O0Q2] There has been an apparatus which satisfies 
the sense of real existence that a counterpart is present 
in front of you and aims at realistic picture communica- 
tion by extracting, for example, a person's picture out of 
the scenery picture of a space in which you are present 
and superimposing the person's picture, a person's pic- 
ture sent from the counterpart, and the picture of a vir- 
tual space to be displayed commonly with a previously- 
stored counterpart on each other and displaying them 
(Japanese Patent Publication No. 4-24914). 
[0003] Particularly, in the case of the prior art, inven- 
tions concerned with acceleration for performing picture 
synthesis and a method for reducing memories are 
made (e.g. Official gazette of Japanese Patent Publica- 
tion No. 5-46592: Picture synthesizer). 
[0004] Though a communication system using picture 
synthesis for synthesizing two-dimensional static pic- 
tures or three-dimensional CQ data has been proposed 
by the prior art, specific discussion on a method for real- 
izing a system for simultaneously synthesizing a plural- 
ity of video (picture) and a plurality of audio and 
displaying them has not been performed from the fol- 
lowing viewpoints. 

[0005] That is, there has been a problem that no spe- 
cific discussion has been performed from the following 
viewpoints: 

(A1) a method for transmitting (communicating and 
broadcasting) and controlling pictures and audio 
under the environment in which data and control 
information (information transmitted by a packet dif- 
ferent from that of data to control the processing of 
terminal side) are independently transmitted by 
using a plurality of logical transmission lines con- 
structed by software on one real transmission line 
or more; 

(A2) a method for dynamically changing header 
information (corresponding to data control informa- 
tion of the present invention) to be added to data for 
a picture or audio to be transmitted; 
(A3) a method for dynamically changing header 
information (corresponding to transmission control 
information of the present invention) to be added for 
transmission; 



(A4) a method for transmitting information by 
dynamically multiplexing and separating a plurality 
of logical transmission lines; 
(A5) a method for transmitting pictures and audio 
5 considering the read and rise periods of program or 
data; and 

(A6) a method for transmitting pictures and audio 
considering zapping. 

10 [0006] However, the method for changing encoding 
systems and a method of discussing data in frames in 
accordance with the frame type of a picture have been 
proposed so far as a method for dynamically adjusting 
the amount of data to be transmitted to a network (H. 

15 Jinzenji and T Tajiri, A study of distributive-adaptive- 
type VOD system, D-81, System Society of Institute of 
Electronics. Information and Communication Engineers 
(lElCE) (1995)). 

[0007] A dynamic throughput scalable algorithm capa- 
20 ble of providing a high-quality video under a restricted 
processing time is proposed as a method for adjusting 
throughput at the encoder side (T. Osako, Y. Yajima, H. 
Kodera, H. Watanabe. K. Shimamura: Encoding of soft- 
ware video using a dynamic throughput scalable algo- 
25 rithm, Thesis Journal of lEICE, D-2, Vol. 80-D-2, No. 2, 
pp. 444-458 (1997)). 

[0008] Moreover, there is an MPEGiyMPEG2 system 
as an example of realizing synchronous reproduction of 
video and audio. 

30 

(B1) The conventional method for discussing a pic- 
ture correspondingly to the frame type of the video 
has a problem that it is difficult to preponderantly 
reproduce an important scene cut synchronously 
35 wHh audio by handling a plurality of vkJeo streams 
or a plurality of audio streams and reflecting the 
intention of an editor because the grading of the 
information which can be handled is in a single 
stream. 

40 (B2) Moreover, it must be possible that a decoder 
decodes every supplied bit stream because it is a 
prerequisite that MPEQiyMPEG2 is realized by 
hardware. Therefore, it is a problem how to corre- 
spond to the case of exceeding the throughput of 

45 the decoder. 

[0009] Moreover, to transmit video, there have been 
some systems including a system such as H. 261 (ITU- 
T Recommendation H. 261 -Video codec for audio-vis- 

50 ual services at px 64) and they have been mounted by 
hardware. Therefore, the case has not occurred that 
decoding Is not completed within a designated time 
because of considering the upper limit of a necessary 
performance when designing hardware. 

55 [0010] The above-designated time denotes a time 
required to transmit a bit stream obtained by coding a 
sheet of video. If decoding is not completed wHhin the 
time, an extra time becomes a delay. If the delay is accu- 
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mulated, the delay from the transmitting side to the 
receiving side increases and the system cannot be used 
as a video telephone. This state must be avoided. 
[0011] Moreover, when decoding cannot be com- 
pleted within a designated time because a communica- 5 
tion counterpart generates an out-of-spec bit stream, a 
problem occurs that a video cannot be transmitted. 
[001 2] The above problem occurs not only for a video 
but also for audio data. 

[001 3] However, in recent years, because the network io 
environment formed by personal computers (PCs) has 
been arranged as the result of spread of internet and 
ISDN, the transmission rate has been improved and it 
has been possible to transmit a video by using PCs and 
a network. Moreover, requests for transmission of video is 
by users have been rapidly increased. Furthermore, a 
video can be completely decoded by software because 
CPU performances have been improved. 
[0014] However, because the same software can be 
executed by personal computers different in structure 20 
such as a CPU. bus width, or accelerator, it is difficult to 
previously consider the upper limit of a necessary per- 
formance and therefore, a problem occurs that a picture 
cannot be decoded within a designated time. 
[001 5] Moreover, when coded data for a video having 25 
a length exceeding the throughput of a receiver Is trans- 
mitted, coding cannot be completed within a designated 
time. 

Problem (CI): Decreasing a delay by decoding a 30 
picture within a designated time. 

When inputting a video as the waveform data of 
claim CI of the present invention or outputting a 
video as the waveform data of claim C7 of the 
present invention as means for solving the problem 3S 
1 , a problem may be left that the substantial work- 
ing efficiency of a transmission line is lowered 
because a part of a transmitted bit stream is not 
used. Moreover, there are some coding systems 
that generate a present decoded video in accord- 4o 
ance with a last decoded picture (e.g. P picture). 
However, because the last decoded picture is not 
completely restored by the means for solving the 
problem 1, there is a problem that deterioration of 
the picture quality influentially increases as time 45 
passes. 

Problem (C2): In the case of the means for solving 
the problem 1 , the substantial working efficiency of 
a transmission line is lowered. Moreover, picture- 
quality deterioration is spread. so 

Furthermore, in the case of mounting by soft- 
ware, the frame rate of a picture is determined by 
the time required for one-time coding. Therefore, 
when the frame rate designated by a user exceeds 
the throughput of a computer, it is impossible to cor- ss 
respond to the designation. 
Problem (C3): When the frame rate designated by a 
user exceeds the throughput of a computer, it is 



impossible to con^espond to the designation. 
Disclosure of the Invention 

[001 6] When conskJering the problems (A1 ) to (A6) of 
the first prior art, it is an object of the present invention 
to provide an audio-vkleo transmitter and audio-video 
receiver and data-processing apparatus and method in 
order to solve at least any one of the problems. 
[0017] Moreover, when conskJering the problems (B1) 
and (B2) of the second prior art, it is another object of 
the present invention to provide data-processing appa- 
ratus and method in order to solve at least one of the 
problems. 

[P018] Furthermore, when considering the problems 
(CI) to (C3) of the last prior art, it is still another object 
of the present invention to provkle waveform-data- 
receiving method and apparatus and waveform-data- 
transmitting method and apparatus, and video-transmit- 
ting m^hod and apparatus and video-receiving method 
and apparatus in order to solve at least one of the prob- 
lems. 

[0019] The present invention according to claim 1 is 
an audio-vkieo transmitting apparatus comprising 

transmitting means for transmitting the content con- 
cerned with a transmitting method and/or the struc- 
ture of data to be transmitted or an identifier 
showing the content as transmission format infor- 
mation through a transmissfon line same as that of 
the data to be transmitted or a transmission line dif- 
ferent from the data transmission line; wherein 
said data to be transmitted is video data and/or 
audio data. 

[0020] The present invention according to claim 2 is 
the audio-video transmitting apparatus according to 
daim 1, wherein said transmission format information is 
included in at least one of data control information 
added to sakl data to control said data, transmission 
control information added to said data to transmit said 
data, and information for corrtrolling the processing of 
the terminal side. 

[0021] The present Invention according to claim 3 is 
the audio-video transmitting apparatus according to 
daim 2, wherein at least one of said data control infor- 
mation, transmission control information, and informa- 
tion for controlling the processing of said terminal side is 
dynamically changed. 

[0022] The present invention according to claim 4 is 
the audio-video transmitting apparatus according to 
daim 3. wherein said data is divided into a plurality of 
packets, and said data control information or said trans- 
mission control information is added not only to the 
head packet of said divided pactete but also to a middle 
packet of them. 

[0023] The present invention according to claim 5 is 
the audio-video transmitting apparatus according to 
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claim 1, wherein an identifier showing whether to use 
timing information concerned with said data as informa- 
tion showing the reproducing time of said data is 
included in said transmission format information. 
[0024] The present invention according to claim 6 is s 
the audio-video transmitting apparatus according to 
claim 1, wherein said transmission format information is 
the structural information of said data and a signal 
which is output from a receiving ^3paratus receiving the 
transnrtltted structural infonmation of said data and which io 
can be received is confirmed and thereafter, said trans- 
mitting means transm'rts corresponding data to said 
receiving apparatus. 

[0025] The present invention according to claim 7 is 
the audio-video transmitting apparatus according to is 
claim 1, wherein said transmission format information 
include (1) an identifier for identifying a program or data 
to be used by a receiving apparatus later and (2) at least 
one of a flag, counter, and timer as information lor 
knowing the point of time in which said program or data 20 
is used or the term of validity for using said program or 
data. 

[0026] The present invention according to claim 8 is 
the audio-video transmitting apparatus according to 
claim 7, wherein said point of time In which said pro- 2s 
gram or data is used is transmitted as transmission con- 
trol information by using a transmission serial number 
for identifying a transmission sequence or as informa- 
tion to be transmitted by a packet different from that of 
data to control terminal-side processing. 30 
[0027] The present invention according to claim 9 is 
the audio-video transmitting apparatus according to 
claim 2 or 3, wherein storing means for storing a plural- 
ity of contents concerned with said transmitting method 
and/or said structure of data to be transmitted and a plu- 3s 
rality of its identifiers are included, and saki identifier is 
included in at least one of said data control information, 
transmission control information, and information for 
controlling terminal-skie processing as said transmis- 
sion format information. 40 
[0028] The present invention according to claim 1 0 is 
the audio-video transmitting apparatus according to 
claim 2 or 3, wherein storing means for storing a plural- 
ity of contents concerned with said transmitting method 
and/or said structure of data to be transmitted are 45 
included, and said contents are included in at least one 
of saki data control information, transmission control 
information, and information for controlling terminal-side 
processing as saki transmission format information. 
[0029] The present invention according to claim 11 is so 
the audio-video transmitting apparatus according to 
claim 1, 2, or 3, wherein a default identifier showing 
whether to change the contents concerned with said 
transmitting method and/or structure of data to be trans- 
mitted is added. 55 
[0030] The present invention according to claim 1 2 is 
the audio-video transmitting apparatus according to 
claim 9, 1 0, or 11 , wherein said identifier or said default 



identifier is added to a predetermined fixed-length 
region of information to be transmitted or said predeter- 
mined position. 

[0031 ] The present invention according to claim 1 3 is 
an audio-vkleo receiving apparatus comprising: receiv- 
ing means for receiving said transmission format infor- 
mation transmitted from the audio-vkieo transmitting 
apparatus of any one of clainns 1 to 12; and transmitted- 
information interpreting means for interpreting said 
received transmisston-format information. 
[0032] The present invention according to claim 1 4 is 
the audio-video receiving apparatus according to claim 
13, wherein storing means for storing a plurality of con- 
tents concerned with sakI transmitting method and/or 
said structure of data to be transmitted and a plurality of 
Its kJentifiers are included, and the contents stored in 
said storing means are used to interpret said transmis- 
sion format information. 

[0033] The present invention according to claim 15 is 
an audio-vkleo transmitting apparatus comprising: infor- 
mation multiplexing means for controlling start and end 
of multiplexing the information for a plurality of logical 
transmission lines for transmitting data and/or control 
information is included; wherein, not only sakI data 
and/or control information multiplexed by said Informa- 
tion multiplexing means but also control contents con- 
cerned with start and end of said multiplexing by said 
information nuiltiplexing means are transmitted as mul- 
tiplexing control information, and said data includes 
video data and/or audio data. 
[0034] The present invention according to claim 1 6 is 
tiie audio-video transmitting apparatus according to 
claim 15, wherein it is possible to select whether to 
transmit said multiplexing control information by arrang- 
ing sakI information without multiplexing it before said 
data and/or control infonnation or transmit said multi- 
plexing control information through a transmission line 
different from the transmission line for transmitting said 
data and/or control information. 
[0035] The present invention according to claim 1 7 is 
an audio-vkleo receiving apparatus comprising: receiv- 
ing means for receiving sad multiplexing control infor- 
mation transmitted from the audio-video transmitting 
apparatus of claim 15 and said multiplexed data and/or 
control information; and separating means for separat- 
ing saki multiplexed data and/or control information in 
accordance with said multiplexing control information. 
[0036] The present invention according to claim 1 8 is 
an audio-video receiving apparatus comprising: main 
looking-listening means for looking at and listening to a 
broadcast program; and auxiliary looking-listening 
means for cyclically detecting the state of a Ixoadcast 
program other than the broadcast program looked and 
listened through said main looking-listening means; 
wherein said detection is performed so that a program 
and/or data necessary when said broadcast program 
looked and listened through sakI main looking-listening 
means is switched to other broadcast program can be 
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smoothly processed, and 

said data includes video data and/or audio data. 
[0037] The present invention according to claim 19 is 
the audio-video transmitting apparatus according to 
claim 1, wherein priority values can be changed in 
accordance with the situation by transmitting the offset 
value of information showing the priority for processing 
of said data. 

[0038] The present invention according to claim 20 is 
an audio-video receiving apparatus comprising: receiv- 
ing means for receiving encoded information to which 
the information concerned witii tiie priority for process- 
ing under an overload state is previously added; and pri- 
ority deciding means for deciding a tiireshoid serving as 
a criterion for selecting whetiier to process an object In 
said information received by said receiving means; 
wherein 

the timing for outputting said received informa- 
tion is compared with the elapsed time after start of 
processing or the timing for decoding said received 
infornrration is compared witii the elapsed time after 
start of processing to change said tiireshoid in accord- 
ance with the comparison result, and video data and/or 
audio data are or is included as said encoding object. 
[0039] The present invention according to claim 21 is 
the audio-video receiving apparatus according to claim 
20, wherein retransmission-request-priority deciding 
means for deciding a threshold serving as a criterion for 
selecting whether to request retransmission of some of 
said information not received because it is lost under 
transmission when it is necessary to retransmit said 
information is included, and 

said decided threshold is decided in accordance 
witii at least one of the priority controlled by said priority 
deciding means, retransmission frequency, lost factor of 
information, insertion interval between in-frame- 
encoded frames, and grading of priorrty 
[0040] The preserrt invention according to claim 22 is 
an audio-video transmitting apparatus comprising: 
retransmission-priority deciding means for dedding a 
threshold serving as a criterion for selecting whether to 
request retransmission of some of said information not 
received because it is lost under transmission when 
retransmission of said unreceived information is 
requested is included, wherein said decided threshold is 
decided in accordance with at least one of tiie priority 
controlled by tiie priority deciding means of said audio- 
video receiving apparatus of claim 20, retransmission 
frequency, lost factor of information, insertion interval 
between in-frame-encoded frames, and grading of prior- 
ity. 

[0041] The present invention according to claim 23 is 
an audio-video transmitting apparatus for transmitting 
said encoded information by using the priority added to 
said encoded information and thereby tiiinning it when 
(1) an actual transfer rate exceeds tiie target transfer 
rate of information for a video or audio or (2) rt is 
decided that writing of said encoded information into a 



transmitting buffer is delayed as the result of comparing 
the elapsed time after start of transmission witii a period 
to be decoded or output added to said encoded informa- 
tion. 

5 [0042] The present invention according to claim 25 is 
a data processing apparatus comprising: receiving 
means for receiving a data series including (1) time- 
series data for audio or video. (2) an inter-time-series- 
data priority showing the priority of the processing 
10 between said time-series-data values, and (3) a plurality 
of in-time-series-data priorities for dividing said time- 
series data value to show the processing priority 
between divided data values; and data processing 
means for performing Processing by using said Inter- 
ns time-series-data priority and said in-time-series-data 
priority together when pluralities of said time-series- 
data values are simultaneously present. 
[0043] The present invention according to claim 27 is 
a data processing apparatus conprising: receiving 
20 means for receiving a data series including (1) time- 
series data for audio or video, (2) an inter-time-series- 
data priority showing the priority of tiie processing 
between said time-series-data values, and (3) a plurality 
of In-time-series-data priorities for dividing said time- 
rs series data value to show the processing priority 
between divided data values; and data processing 
means for distributing throughput to each of said time- 
series-data values in accordance with said inter-time- 
series-data priority and moreover, adaptively deteriorat- 
30 ing the processing quality of the divided data in said 
time-series data In accordance with said in-time-series- 
data priority so tiiat each of said time-series-data values 
is kept within said distributed throughput. 
[0044] The present invention according to claim 29 is 
35 a data processing apparatus characterized by, when an 
in-time-series-data priority for a video is added every 
frame of said video and said video for each frame is 
divided into a plurality of packets, adding said in-time- 
series-data priority only to tiie header portion of a 
40 packet for transmitting the head portion of a frame of 
said video accessible as independent information. 
[0045] The present invention according to claim 31 is 
the data processing apparatus according to any one of 
claims 25. 27, and 29, wherein said in-time-series-data 
45 priority is described in the header of a packet to perform 
priority processing. 

[0046] The present invention according to claim 33 is 
the data processing apparatus according to any one of 
claims 25, 27, and 29, wherein the range of a value 

50 capat)le of expressing said in-time-seri es-data priority is 
made variable to perform priority processing. 
[0047] The present Invention according to claim 34 is 
a data processing method comprising the steps of: 
inputting a data series including time-series data for 

55 audio or video and an inter-time-series-data priority 
showing tiie processing priority between said time- 
series data values; and 

processing priorities by using said inter-time- 
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series-data priority as the value of a relative or absolute 
priority. 

[0048] The present invention according to daim 36 is 
a data processing method comprising the steps of: clas- 
sifying time-series data values for audio or video; input- 
ting a data series including said time-series data and a 
plurality of in-time-series-data priorities showing the 
processing priority between said classified data values; 
and processing priorities by using said in-time-series- 
data priority as the value of a relative or absolute prior- 
ity. 

[0049] Moreover, to solve tiie problem (CI), the 
present invention is characterized by: 

inputting, for example, a video as waveform data in 
accordance with the wavdbrm-data-transmitting 
method of claim 63; or 

outputting. for example, a video as waveform data 
in accordance witii the waveform-data-receiving 
nfiethod of claim 69. 

[0050] Moreover, to solve tiie problem (02). the 
present invention is characterized by: 

(d) outputting tiie execution time of each group 
obtained tiirough estimation in accordance witii the 
waveform-data-receiving metiiod of claim 69;or 

(d) inputting a data string constituted with the exe- 
cution time of each group; and 

(e) computing tiie execution frequency of each 
group for completing decoding witiiin a time 
required to transmit a code length determined by 
the designation of a rate controller or the like in 
accordance with each execution time of tiie receiv- 
ing means in accordance witii the wave-data-trans- 
mitting metiiod of claim 63. 

[0051] Furttiermore, to solve the problem (03). the 
present invention is characterized by: 

(d) estimating the execution time of each group in 
accordance with the processing time required to 
encode a video and each execution frequency out- 
put by counting means; and 

(e) estimating the processing time required to 
encode a video by using tiie above execution time 
and computing the execution frequency of each 
group in which the processing time does not 
exceed a time usable to process one sheet of pic- 
ture determined by a frame rate given as the desig- 
nation of a user in accordance with tiie waveform- 
data-tiBnsmitting method of daim 67. 

[0052] The present invention has the above structure 
to obtain the execution frequency of indispensable 
processing and that of dispensable processing, transmit 
the execution frequencies to the receiving side, and 
estimate the time required for each processing in 



accordance with tiie execution frequencies and the 
decoding time. 

[0053] By redudng each execution frequency of dis- 
pensable processing so tiiat tiie time required for 

5 decoding becomes shorter tiian a designated time in 
accordance with the estimated time of each processing, 
it is possible to control the decoding time to the desig- 
nated time or shorter and keep a delay small. 
[0054] Olaims 67 and 73 are mainly listed as the 

10 inventions for solving the problem (01). 

[0055] Moreover, it is possible to set tiie decoding exe- 
cution time to a value equal to or less tiian a designated 
time by transmitting the execution time of indispensable 
processing and tiiat of dispensable processing esti- 

15 mated by the receiving side to tiie transmitting side and 
determining each execution frequency at the transmit- 
ting side in accordance witii each execution time. 
[0056] Olaims 75 and 77 are mainly listed as the 
inventions for solving the problem (02). 

20 [0057] Moreover, it is possible to set tiie encoding esti- 
mation time to a value equal to or less than a user des- 
ignated time by estimating the execution time of 
indispensable processing and tiiat of dispensable 
processing and determining each execution frequency 

25 in accordance witii each execution time and tiie user 
designated time determined by a frame rate designated 
by a user. 

[0058] Olalm 79 is mainly listed as tiie invention for 
solving tiie problem (03). 

30 

Brief Description of tiie Drawings 
[0059] 

35 Rgure 1 is a schematic block diagram of the audio- 
video transceiver of an embodiment of the present 
invention; 

Rgure 2 is an illustration showing a reception con- 
trol section and a separating section; 
40 Rgure 3 is an illustafation shoving a method for 
transmitting and controlling video and audio by 
using a plurality of logical transmission lines; 
Rgure 4 is an illustration showing a method for 
dynamically changing header information added to 
45 the data for a video or audio to be transmitted; 

Rgures 5(a) and 5(b) are illustrations showing a 
metiiod for adding AL information; 
Rgures 6(a) to 6(d) are illustrations showing exam- 
ples of a method for adding AL information; 
50 Rgure 7 is an illustration showing a method for 
transmitting information by dynamically multiplexing 
and separating a plurality of logical transmission 
lines; 

Rgure 8 is an illustration showing a procedure for 
55 transmitting a broadcasting program; 

Rgure 9(a) is an illustration showing a method for 
transmitting a video or audio considering the read 
and rise time of program or data when the program 
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or data is present at a receiving terminal; 
Figure 9(b) is an illustration showing a method for 
transmitting a video or audio considering the read 
and rise time of program or data when the program 
or data is transmitted; s 
Figure 10(a) is an illustration showing a method for 
corresponding to zapping; 
Rgure 10(b) is an illustration showing a method for 
conresponding to zapping; 

Rgure 11(a) is an illustration showing a specific 10 
example of the protocol to be actually transferred 
between terminals; 

Figure 11(b) is an illustration showing a specific 
example of the protocol to be actually transferred 
between terminals; is 
Figure 12 is an illustration showing a specific exam- 
ple of the protocol to be actually transferred 
between terminals; 

Figure 13(a) is an illustration showing a specific 
ecample of the protocol to be actually transfen^ed 20 
between terminals; 

Figure 13(b) Is an illustration showing a specific 
example of the protocol to be actually transfenred 
between terminals; 

Figure 13(c) is an illustration showing a specific 25 
example of the protocol to be actually transfenred 
between terminals; 

Figure 14 is an Illustration showing a specific exam- 
ple of the protocol to be actually transferred 
between terminals; 30 
Rgure 15 is an illustration showing a specific exam- 
ple of the protocol to be actually transfenred 
between terminals; 

Figure 16(a) is an illustration showing a specific 
example of the protocol to be actually transfenred 35 
between terminals; 

Figure 16(b) is an illustration showing a specific 
example of the protocol to be actually transfenred 
between terminals; 

Figure 1 7 is an illustration showing a specific exam- 40 
pie of the protocol to be actually transfenred 
between terminals; 

Figure 18 is an illustration showing a specific exam- 
ple of the protocol to be actually transfenred 
between terminals; 45 
Figure 19(a) is an illustration showing a specific 
example of the protocol to be actually transferred 
between terminals; 

Figure 19(b) is an Illustration showing a specific 
example of the protocol to be actually transferred so 
between terminals; 

Figures 20(a) to 20(c) are block diagrams of dem- 
onstration systems of CGD of the present invention; 
Figure 21 is an illustration showing a method for 
adding a priority under overload at an encoder; 55 
Figure 22 is an illustration describing a method for 
deciding a priority at a receiving terminal under 
overload; 



Rgure 23 is an illustration showing temporal 
change of priorities; 

Rgure 24 is an illustration showing stream priority 
and object priority; 

Rgure 25 is a schematic block diagram of a video 
encoder and a video decoder of an enrtxxiiment of 
the present invention; 

Rgure 26 is a schematic block diagram of an audio 
encoder and an audio decoder of an embodiment of 
the present invention; 

Rgures 27(a) and 27(b) are illustrations showing a 
priority adding section and a priority deciding sec- 
tion for controlling the priority of processing under 
overload; 

Figures 28(a) to 28(c) are illustrations showing the 
grading for adding a priority; 
Rgure 29 is an illustration showing a method for 
assigning a priority to multi-resolution video data; 
Rgure 30 is an illustration showing a method for 
constituting a communication payload; 
Rgure 31 is an illustration showing a metiiod for 
making data correspond to a communication pay- 
load; 

Rgure 32 is an illustration showing tiie relation 
between object priority, stream priority, and commu- 
nication packet priority; 

Rgure 33 is a block diagram of a transmitter of the 
first embodiment of tiie present invention; 
Rgure 34 is an illustration of tiie first embodiment; 
Rgure 35 is a block diagram of tiie receiver of the 
third embodiment of the present invention; 
Rgure 36 is a block diagram of the receiver of the 
fifth embodiment of the present Invention; 
Rgure 37 is an illustration of tiie fifth embodiment; 
Rgure 38 is a block diagram of tiie transmitter of the 
sixth embodiment of tiie present invention; 
Rgure 39 is a block diagram of tiie transmitter of tiie 
eighth embodiment of tiie present invention; 
Rgure 40 is a flowchart of tiie transmission method 
of the second embodiment of the present invention; 
Rgure 41 is a flowchart of tiie reception method of 
the fourth embodiment of the present inventfon; 
Rgure 42 is a flowchart of the transmission method 
of tiie seventh embodiment of the present inven- 
tion; 

Figure 43 is a flowchart of the transmission metfiod 
of the ninfli embodiment of the present invention; 
Rgure 44 is a block diagram showing an audio- 
video transmitter of the present invention; 
Rgure 45 is a block diagram showing an audio- 
video receiver of the present invention; 
Rgure 46 is an illustration for explaining priority 
adding means for adding a priority to a video and 
audio of an audio-video transmitter of the present 
invention; and 

Figure 47 is an illustration for explaining priority 
deciding means for deckiing whetiier to perform 
decoding by interpreting the priority added to a 
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video and audio of an audio-video receiver of the 
present invention. 

(Description of Symbols) 
[0060] 

1 1 Reception control section 

12 Separating section 

13 Transmitting section 

14 Video extending section (Picture extending sec- 
tion) 

15 Video-extension control section (Picture-ecten- 
slon control section) 

16 Video syntiiesizing section (Picture syntiieslzing 
section) 

17 Output section 

18 Terminal control section 

401 1 Transmission control section 

4012 Video encoding section (Picture encoding 
section) 

4013 Reception control section 

4014 Video decoding section (Picture decoding 
section) 

401 5 Video synthesizing section (Picture synthesiz- 
ing section) 

401 6 Output section 

4101 Video encoder (Picture encoder) 

4102 Video decoder (Picture decoder) 

301 Receiving means 

302 Estimating means 

303 Video decoder (i.e. Dynamic-picture or Moving 
picture decoder) 

304 Frequency reducing means 

306 Output terminal 

307 Input terminal 

3031 Variable decoding means 

3032 Inverse orthogonal transforming means 

3033 Switching unit 

3034 Movement compensating means 

3035 Execution-time measuring means 

Best Mode lor Canrying Out tiie Invention 

[0061] Embodiments of the present invention are 
described below by referring to the accompanying draw- 
ings. 

[0062] The embodiments described below mainly 
solve any one of the above problems (A1) to (AS). 
[0063] A "picture (or video)" used for tiie present 
invention includes a static-picture and a moving-picture. 
Moreover, a purposed picture can be a two-dimensional 
picture like computer graphics (CG) or three-dimen- 
sional picture data constituted with a wire-frame model. 
[0064] Figure 1 is a schematic block diagram of the 
audio-video transceiver of an embodiment of the 
present invention. 

[0065] In Figure 1 , a reception control section 1 1 for 



receiving information and a transmitting section 13 for 
transmitting Information are information transmitting 
means such as a coaxial cable. CATV, LAN, and 
modem. Communication environment can be the envi- 
ronment in which a plurality of logical transmission lines 
can be used without considering multiplexing means 
such as internet or the environment in which multiplex- 
ing means must be considered such as analog tele- 
phone or satellite broadcast. 
[0066] Moreover, a system for bidirectionally transfer- 
ring video and audio between terminals such as a pic- 
ture telephone or teleconference system or a system for 
broadcasting broadcast-type video and audio through 
satellite broadcast, CATV, or internet are listed as termi- 
nal connection systems. The present invention tates 
such terminal connection systems into consideration. 
[0067] A separating section 12 shown in Figure 1 is 
means for analyzing received Information and separat- 
ing data from control information. Specifically, tiie sec- 
tion 12 is means for decomposing tiie header 
information for transmission added to data and data or 
decomposing the header for data control added to tiie 
data and tiie contents of tiie data. A picture extending 
section 14 is means for extending a received video. For 
example, a video to be extended can be the com- 
pressed picture of a standardized moving(dynamic) or 
static picture such as H.261 , H.263. MPEG1/2, or JPEG 
or not. 

[0068] The picture-extension control section 1 5 shown 
in Rgure 1 Is means for nmnitoring tiie extended state 
of a video. For example, by monitoring the extended 
state of a picture, it is possible to empty-read a receiving 
buffer without extending the picture when the receiving 
buffer almost causes overflow and restart the extension 
of the picture after the picture is ready for ectension. 
KK)69] Moreover, in Figure 1, a picture syntiiesizing 
section 16 is means for synthesizing an extended pic- 
ture. A picture synthesizing method can be defined by 
describing a picture and its structural information (dis- 
play position and display time (moreover, a display 
period can be included)), a method for grouping pic- 
tures, a picture display layer (depth), an object ID 
(SSRC to be described later), and tiie relation between 
attributes of tiiem with a script language such as JAVA, 
VRML, or MHEG. The script describing flie syntiiesizing 
method is input or output through a network or a local 
memory 

[0070] Moreover, an output section 17 Is a display or 
printer for outputting a picture syntiiesized result. A ter- 
minal control section 18 Is means for controlling each 
section. Furtiiermore, it is possible to use a structure for 
extending an audio instead of a picture (it Is possikMe to 
constitute the structure by changing a picture extending 
section to an audio extending section, a picture exten- 
sion control section to an audio extension control sec- 
tion, and a picture synthesizing section to an audio 
syntiiesizing section) or a structure for extending a pic- 
ture and an audio and syntiiesizing and displaying them 
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while keeping temporal synchronization. 
[0071 ] Furthermore, it is possible to transmit a picture 
and an audio by using a picture compressing section for 
compressing a picture, a picture compression control 
section for controlling the picture compressing section, 
an audio compressing section for compressing an 
audio, and an audio compression control section for 
controlling the audio compressing section. 
[0072] Figure 2 is an illustration showing a reception 
control section and a separating section. 
[0073] By constituting tiie reception control section 1 1 
shown in Figure 1 with a data receiving section 101 for 
receiving data and a control information receiving sec- 
tion 102 for receiving the control information for control- 
ling data and tiie separating section 12 witii a 
transmission format storing section 103 for storing a 
transmission structure (to be described later in detail) 
for interpreting transmission contents and a transmis- 
sion information interpreting section 104 for interpreting 
transmission contents in accordance witii tiie transmis- 
sion structure stored in the transmission format storing 
section 103, It is possible to independently receive data 
and control Information. Therefore, for example, it is 
easy to delete or move a received video or audio while 
receiving it. 

[0074] As described above, it is possible for the com- 
munication environment purposed by tiie reception con- 
trol section 11 to use a communication environment 
(internet profile) in which a plurality of logical transmis- 
sion lines can be used without considering multiplexing 
means like intern^ or a comnrtunlcation environment 
(Raw profile) in which multiplexing means must be con- 
sidered like analog telephone or satellite broadcast, 
l-lowever, a user premises a communication environ- 
ment in which a plurality of logical transmission lines 
(logical channels) are prepared (for example, in the 
case of a communication environment in which TCP/IP 
can be used, the expression referred to as "communica- 
tion port" Is generally used). 

[0075] Moreover, as shown in Figure 2, it is assumed 
that the reception control section 11 receives one type 
of data transmission line or more and one type of control 
logical transmission line for controlling data to be trans- 
mitted or more. It is also possible to prepare a plurality 
of transmission lines for transmitting data and only one 
transmission line for controlling data. Moreover, it is 
possible to prepare a transmission line for controlling 
data every data transmission like the RTP/RTCP also 
used for H.323. Furthermore, when considering the 
broadcast using UDP, it is possible to use a communica- 
tion system using a single communication port (multi- 
cast address). 

[0076] Figure 3 Is an illustration for explaining a 
method for transmitting and controlling video and audio 
by using a plurality of logical transmission lines. The 
data to be transmitted is referred to as ES (Elementary 
Stream), wl^ch can be picture information for one frame 
or picture information in QOBs or macroblocks smaller 



tiian one frame in the case of a picture. 
[0077] In tiie case of an audio, it is possible to use a 
fixed lengUi decided by a user. Moreover, tiie data-con- 
trol header information added to tiie data to be transmit- 

5 ted is referred to as AL (Adaptation Layer information). 
The information showing whether it is a start position 
capable of processing data, information showing data- 
reproducing time, and Information showing tiie priority 
of data processing are listed as tiie AL information. Data 

10 control information of the present invention corresponds 
to the AL information. Moreover, It is not always neces- 
sary for the ES and AL used for the present invention to 
coincide with tiie contents defined by MPEG1/2. 
[0078] The information showing whether it is a start 

75 position capable of processing data specifically includes 
too types of information. First one is a flag for random 
access, that is, the information showing tiiat it can be 
individually read and reproduced independently of pre- 
ceding or following data such as intra-frame (I picture) in 

20 tiie case of a picture. Second one Is the information 
capable of defining an access flag as a flag for showing 
tiiat it can be individually read, that is, the information 
showing that it is the head of pictures in GOBs or mac- 
roblocks in the case of a picture. Therefore, absence of 

25 an access flag shows tiie middle of data. Botii random 
access flag and access flag are not always necessary 
as the information showing that it Is a start position 
capable of processing data. 

[0079] There is a case in which no problem occurs 
30 even if both the flags are not added In tiie case of tiie 
real time communication such as a teleconference sys- 
tem. However, to simply perform edition, a random 
access flag is necessary It Is also possible to decide 
whetiier a flag is necessary or which flag is necessary 
35 through a communication channel before transferring 
data. 

[0080] The information indicating a data reproducing 
time shows the information for time synchronization 
when a picture and an audio are reproduced, which is 

40 referred to as PTS (Presentation Time Stamp) in the 
case of MEPGiy2. Because time synchronisation is not 
normally considered in tiie case of the real time commu- 
nication such as a teleconference system, the informa- 
tion representing a reproducing time is not always 

45 necessary. The time interval between encoded frames 
may be necessary information. 
[0081] By making the receiving side adjust a time 
interval, it is possible to prevent a large fluctuation of 
frame Intervals. However, by making the receiving side 

50 adjust the reproducing interval, a delay may occur. 
Therefore, it may be decided that tiie time information 
showing the frame interval between encoded frames is 
unnecessary 

[0082] To decide whetiier the Information showing a 
55 data reproducing time represents a PTS or frame Inter- 
val, it is also possible to decide tiiat the data reproduc- 
ing time is not added to data before transmitting the 
data and communicate the decision to a receiving termi- 
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nal through the communication channel and transmit 
the data together with decided data control information. 
[0083] When the information showing the priority for 
processing data cannot be processed or transmitted 
due to the load of a receiving terminal or that of a net- 
work, rt Is possible to reduce the load of the receiving 
terminal or network by stopping the processing or trans- 
mission of the data. 

[0084] The receiving terminal is able to process the 
data with the picture-extension control section 15 and 
the network is able to process the data with a relay ter- 
minal or router. The priority can be expressed by a 
numerical value or a flag. Moreover, by transmitting the 
offset value of the information showing the data- 
processing priority ses control information or data control 
information (AL information) together with data and add- 
ing tiie offset value to the priority previously assigned to 
a video or audio in the case of a sudden fluctuation of 
the load of a receiving terminal or network, it is possible 
to set a dynamfo priority con^esponding to the operation 
state of a system. 

[0085] Furthermore, by transmitting the information for 
identifying presence/absence of scramble, pres- 
ence/absence of copyright, and original or copy as con- 
trol information together with a data identifier (SSRC) 
separately from data as control information, it is simpli- 
fied to cancel the scramble at a relay node. 
[0086] Moreover, tiie information showing tiie data 
processing priority can be added every stream consti- 
tuted witii the aggregation of frames of a plurality of pic- 
tures or audios or every frame of video or audio. 
[0087] Priority adding means for deciding the 
encoded-information processing priority under overload 
in accordance witii tiie predetermined rules by the 
encocfing metiiod such as H.263 or G.723 and making 
the encoded information conrespond tothe decided pri- 
ority is provided for a transmitting terminal unit (see Fig- 
ure 46). 

[0088] Figure 46 is an illustration for explaining priority 
adding means 5201 for adding a priority to a picture and 
an audio. 

[0089] That is, as shown in Rgure 46, a priority is 
added to encoded-video data (to be processed by video 
encoding means 5202) and encoded-audio data (to be 
processed by audio encoding means 5203) in accord- 
ance with predetermined rules. The rules for adding pri- 
orities are stored in priority adding rules 5204. The rules 
include rules for adding a priority higher than that of a P- 
frame (irrter-frame encoded picture frame) to an l-frame 
(intra-frame encoded picture frame) and rules for adding 
a priority lower than tiiat of an audio to a picture. More- 
over, it is possible to change the rules in accordance 
witii the designation of a user. 
[0090] Priority-adding objects are scene changes in 
the case of a picture or an audio block and audioless 
block in the case of a picture frame, stream, or audio 
designated by an editor or user. 
[0091 ] To add a priority in picture or audio frames for 



defining the processing prior'rty under overload, tiie fol- 
lowing methods are considered: a method for adding a 
priority to a communication header and a method for 
embedding a priority in tiie header of a bit stream in 

5 which a video or audio is encoded under encoding. The 
former makes it possible to obtain tiie information for 
priority without decoding it and the latter makes it possi- 
ble to independently handle a single bit stream witiiout 
depending on a system. 

10 [0092] When one picture frame (e.g. intra-frame 
encoded l-frame or inter-frame encoded P- or B-frame) 
is divided into a plurality of transmission packets, a pri- 
ority is added only to a communication header for trans- 
mitting tiie head of a picture frame accessible as 

IS Independent information in tiie case of a picture (when 
priorities are equal in the same picture frame, it is pos- 
sible to assume tiiat tiie priorities are not changed 
before the head of the next accessible picture frame 
appears). 

20 [0093] Moreover, it is possible to realize configuration 
in accordance with control information by making the 
range of a value capable of expressing a priority varia- 
ble (for sample, expressing time information witii 16 
bits or 32 bits depending on tiie purpose). 

25 [0094] Furtiiemtore, in tiie case of a decoder, priority 
deciding means for deciding a processing method is 
provided for a receiving terminal unit in accordance witii 
tiie priority under overload of received various encoded 
pieces of information (see Figure 47). 

30 [0095] Figure 47 is an illustration for interpreting Prior- 
ities added to a picture and an audio and explaining pri- 
ority deciding means 5301 for deciding whether to 
perform decoding. 

[0096] That is, as shown In Figure 47, the priorities 
35 include a priority added to each stream of each picture 
or audio and a priority added to each frame of a picture 
or audio. It is possible to use tiiese priorities independ- 
ently or by making a frame priority correspond to a 
stream priority. The priority deciding means 5301 
40 decides a stream or frame to be decoded in accordance 
with tiiese priorities. 

[0097] Decoding is performed by using two types of 
priorities for deciding a processing priority under over- 
load at a terminal. 

45 [0098] That is. a stream priority (inter-time-series pri- 
ority) for defining a relative priority between bit streams 
such as a picture and audio and a frame priority (intra- 
time-series priority) for defining a relative priority 
between decoding units such as picture frames in the 

50 same stream are defined (Figure 24). 

[0099] The former stream priority makes it possible to 
handle a plurality of videos or audios. The latter frame 
priority makes it possible to change scenes or add dif- 
ferent priorities even to the same intra-frame encoded 

55 picture frames (l-frame) in accordance with the intention 
of an editor. 

[0100] By making a stream priority correspond to a 
time assigned to an operating system (OS) for encoding 
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or decoding a picture or audio or a processing priority 
and thereby controlling the stream priority, it is possible 
to control a processing time at an OS level. For exam- 
ple, in the case of Windows95/NT of Microsoft Corpora- 
tion, a priority can be defined at five OS levels. By 5 
realizing encoding or decoding means by software in 
threads, it is possible to decide a priority at an OS level 
to be assigned to each thread in accordance with the 
stream priority of a purposed stream. 
[0101] The frame priority and stream priority 
described above can be applied to a transmission 
medium or data-recording medium. For example, by 
defining the priority of a packet to be transmitted as an 
access unit priority, it is possible to dedde a priority con- 
cerned with packet transmission or a priority for 
processing by a terminal under overioad in accordance 
with the relation between frame priority and stream pri- 
ority such as the relation of Access Unit Priority = 
Stream Priority - Frame Priority 
[0102] Moreover, it is possible to decide a priority by 
using a floppy disk or optical disk as a data-recording 
medium. Furthermore, it is possible to decide a priority 
by using not only a recording medium but also an object 
capable of recording a program such as an IC card or 
ROM cassette. Furthermore, it is possible to use a 
repeater for a picture or audio such as a router or gate- 
way for relaying data. 

[0103] As a specific method for using a priority, when 
a receiving terminal is overloaded, priority deciding 
means for deciding the threshokl of the priority of 
encoded information to be processed is set to a picture- 
extension control section or audio-extension control 
section and the time to be displayed (PTS) is compared 
with the elapsed time after start of processing or the 
time to be decoded (DTS) is conrpared with the time 
elapsed time after start of processing to change tiiresh- 
olds of the priority of encoded information to be proc- 
essed In accordance with the comparison result (it is 
also possible to refer to the insertion interval of l-frame 
or the grading of a priority as the information for chang- 
ing tiiresholds). 

[0104] In tiie case of the example shown in Figure 
20(a), a picture with the size of captured QCIF or GIF is 
encoded by an encoder (H.263) under encoding to out- 
put a time stamp (PTS) showing the time for decoding 
(DTS) or tiie time for displaying the picture, priority infor- 
mation showing processing sequence under overioad 
(CGD, Computational Graceful Degradation), frame 
type (SN), and sequence number together with 
encoded information. 

[01 05] Moreover, in the case of the example shown in 
Figure 20(b), an audio is also recorded through a micro- 
phone and encoded by an encoder (G.721) to output a 
time stamp (PTS) showing tfie time for decoding (DTS) 
or the time for reproducing an audio, priority information 
(CGD), and sequence number (SN) together with 
encoded information. 

[0106] Under decoding, as shown in Figure 20(c), a 



picture and an audio are supplied to separate buffers to 
compare tiieir respective DTS (decoding time) with the 
elapsed time after start of processing. When DTS is not 
delayed, the picture and the audio are supplied to tiieir 
con-esponding decoders (H.263 and G.721). 
[0107] The example in Figure 21 describes a method 
for adding a priority by an encoder under overload. For 
a picture, high priorities of "0" and "1" are assigned to I- 
frame (intra-frame encoded picture frame) (the smaller 
a numerical becomes, the lower a priority becomes). P- 
frame has a priority of "2" which is lower than that of I- 
frame. Because two levels of priorities are assigned to I- 
frame. it is possible to reproduce only l-frame having a 
priority of "0" when a terminal for decoding has a large 
load. Moreover, it is necessary to adjust the insertion 
interval of l-frame in accordance with a priority adding 
metiiod. 

[0108] The example in Figure 22 shows an illustration 
showing a method for deciding a priority at a receiving 
terminal under overload. The priority of a frame to be 
disused is set to a value larger than a cutOffPriority 
That is, every picture frame is assumed as an object to 
be processed. It is possible to previously know the max- 
imum value of priorities added to picture frames by com- 
municating it from the transmitting side to the receiving 
side (step 101). 

[01 09] When DTS is compared witii the elapsed time 
after start of processing and resultantly, the elapsed 
time is larger tiian DTS (when decoding is not in time), 
tiie threshokJ of tiie priority of a pfoture or audio to be 
processed is decreased to thin out processings (step 
102). However, when the elapsed time after start of 
processing is smaller than DTS (decoding is in time), 
the threshold of a priority is increased in order to 
increase tiie number of pictures or audio which can be 
processed (step 103). 

[01 1 0] If the image from one before is skipped by P- 
frame, no processing is performed. If not, a priority off- 
set value is added to the priority of a picture frame (or 
audio frame) to compare the priority offset value with 
tiie threshold of tiie priority When tiie offset value does 
not exceed tiie threshold, data to be decoded is sup- 
plied to a decoder (step 104). 
[01 1 1 ] A priority offset allows the usage of previously 
checking the performance of a machine and communi- 
cating tiie offset to a receiving terminal (it is also possi- 
ble that a user issues designation at tiie receiving 
terminal) and the usage of changing priorities of a plu- 
rality of video and audio streams in streams (for exam- 
ple, thinning out processings by increasing the offset 
value of tiie rearmost background). 
[0112] When a multi-stream is purposed, it is also 
possible to add a priority for each stream and decide the 
skip of decoding of a picture or audio. Moreover, in the 
case of real time communication, it is possible to decide 
whetiier decoding is advanced or delayed at tiie termi- 
nal by handling the TR (Temporary Reference) of H.263 
similarly to DTS and realize the skipping same as 
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described above. 

[0113] Figure 23 is an Illustration showing temporal 
change of priorities by using the above algorithm. 
[01 1 4] Rgure 23 shows the change of a priority to be 
added to a picture frame. This priority is a priority for 
deciding whether to perform decoding when a terminal 
is overloaded, which is added every frame. The smaller 
the value of a priority becomes, the higher the priority 
becomes. In the case of the example in Figure 23, 0 has 
the highest priority. When the threshold of a priority is 3, 
a frame having a priority to which a value larger than 3 
is added is disused without being decoded and a frame 
having a priority to which a value of 3 or less Is added is 
decoded. By selectively discussing frames In accord- 
ance with priorities, it is possible to control the load of a 
terminal. It is also possible to dynamically decide the 
priority threshold in accordance with the relation 
between the present processing time and the decoding 
time (DTS) to be added to each frame. This technique 
can be applied not only to a picture frame but also to an 
audio in accordance with the same procedure. 
[01 1 5] In the case of a transmission line such as inter- 
net, when it is necessary to retransmit encoded infor- 
mation lost under transmission, it is possible to 
retransmit only a picture or audio required by the receiv- 
irig side by providing a retransmission request priority 
deciding section for dedding the threshold of the priority 
of the encoded information to be retransmitted for a 
reception control section and deciding the threshold of 
the priority added to the encoded information whose 
retransmission should be requested in accordance with 
the information for priority, retransmission frequency, 
loss rate of information, insertion interval of intra-frame 
encoded frame, grading of priority (e.g. five-level prior- 
ity) which are controlled by the priority deciding section. 
If the retransmission frequency or loss rate of informa- 
tion is too large, it is necessary to raise the priority of the 
information to be retransmitted and lower the retrans- 
mission or loss rate. Moreover, by knowing the priority 
used for the priority deciding section, it is possible to 
prevent the information to be processed from being 
transmitted. 

[01 1 6] In the case of a transmitting terminal, when an 
actual transfer rate exceeds the target transfer rate of 
the information of the transmitting terminal or when writ- 
ing of the encoded information into a transmitting buffer 
is delayed as the result of comparing the elapsed time 
after start of transfer processing with the time added to 
the encoded information to be decoded or displayed, it 
is possible to transmit a picture or audio matching with 
the target rate by using a priority added to encoded 
Irrformation and used by the priority deciding section of 
the receiving terminal when the terminal is overioaded 
and thereby thinning out transmissions of information. 
Moreover, by introducing the processing skipping func- 
tion under overload performed at the receiving-side ter- 
minal into the transmitting-side terminal, it is possible to 
control a feilure due to overioad of the transmitting-side 



terminal. 

[0117] By making it possible to transmit only neces- 
sary information out of the above-described AL informa- 
tion according to necessity, it is possible to adjust the 

5 amount of information to be transmitted to a narrow- 
band communication channel such as an analog tele- 
phone line. It is possible to recombine the AL informa- 
tion (data control information) used for the transmitting 
side by deciding the data control information to be 

10 added to data at a transmitting-side terminal before 
transmitting the data, communicating the data control 
information to be used to a receiving terminal as control 
information (for example, using only a random access 
flag), and rewriting at the receiving-side terminal based 

IS on the obtained control information tiie Information 
about a transmission structure (showing which AL infor- 
mation is used) stored in the transmission format storing 
section 103 (see Figure 16). 

[0118] Figure 4 is an illustration for explaining a 
20 method for dynamically changing header information 
added to tiie data for a picture or audio to be transmit- 
ted. In the case of the example in Figure 4, the data 
(ES) to be transmitted is decomposed into data pieces 
and the identifying information (sequence number) for 
25 showing the sequence of data, the information (marter 
bit) showing whether it is a start position capable of 
processing data pieces, and time information (time 
stamp) concerned witii transfer of data pieces are 
added to data pieces in the form of communication 
30 headers by assuming tiiat the above pieces of informa- 
tion correspond to transmission control information of 
the present invention. 

[0119] Specifically. RTP (Realtime Transfer Protocol, 
REC1889) uses the information for the above sequence 

35 number, martor bit, time stamp, object ID (referred to as 
SSRC), and version number as communication head- 
ers. Though a header-information item can be 
extended, the above items are always added as fixed 
items. However, when the realtime communication such 

40 as the case of a video telephone and transmission of 
accumulated media such as the case of video-on- 
demand are present together in an environment in 
which a plurality of different encoded pictures or audio 
are simultaneously transmitted, identifying means is 

45 necessary because meanings of confimunication head- 
ers are different from each otiier. 
[0120] For example, time-stamp information shows 
PTS that Is a reproducing time as previously described 
in the case of MPEGI/S. In the case of H.261 or H.263. 

60 however, the time-stamp information shows a time inter- 
val when the Information is encoded. However, to proc- 
ess H.263 synchronously with an audio, it is necessary 
to show that a time stamp is PTS information. This is 
because time-stamp information shows the time interval 

55 between encoded frames in the case of H. 263 and it is 
defined by RTP that tiie time stamp of the first frame is 
random. 

[01 21 ] Therefore, it is necessary to add a flag showing 
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whether a time stamp is PTS as (a) communication 
header information (it is necessary to extend a commu- 
nication header) or (b) header information for payload of 
H.263 or H.261 (that is, AL information) (in this case. It 
is necessary to extend payload information). 
[0122] A marker bit serving as the information show- 
ing whether it is a start position capable of processing 
data pieces is added as RIP header information. More- 
over, as described above, there is a case In which it is 
necessary to provide an access flag showing that It Is a 
start position capable of accessing data and a random 
access flag showing that It Is possible to access data at 
random for AL information. Because doubly providing 
flags for a communication header lowers the efficiency, 
a method of substituting an AL flag by a flag prepared 
for the communication header is also considered. 

(c) The problem is solved by newly providing a flag 
showing that an AL flag is substituted by tiie header 
added to a communication header witiiout adding a 
flag to AL for the communication header or defining 
that tiie marker bit of the communication header is 
the same as that of AL (it is expected that interpre- 
tation can be quickly performed compared to the 
case of providing a flag for AL). That Is, a flag is 
used which shows whether the marker bit has the 
same meaning as the flag of AL. in this case, it is 
considered to improve tiie communication header 
or describe it in an extension region. 

[0123] However, (d) it is also possible to interpret tiie 
meaning of the marker bit of the communication header 
so as to mean tiiat at least either of a random access 
flag and an access flag is present in AL. In tiiis case, it 
Is possible to know that the meaning of Interpretation is 
changed from tiie conventional case by tiie version 
number of the communication header. Moreover, 
processing is simplified by providing an access flag or 
random access flag only for the communication header 
or the header of AL (for the former, a case of providing 
the flag for both tiie headers is conskfered but it is nec- 
essary to newly extend the communication header). 
[0124] It is already described to add the information 
showing the priority of data processing as the informa- 
tion for AL. By adding the dataiDrocessing priority to the 
communication header, it is possible to decide the 
processing of tiie data-processing priority witiiout inter- 
preting the contents of data also on a network. Moreo- 
ver, in tiie case of IPv6, it is possible to add the priority 
at a layer lower than the level of RTR 
[0125] By adding a timer or counter for showing the 
effective period of data processing to the communica- 
tion header of RTP, it is possible to decide how tiie state 
of a transmitted packet changes. For example, when 
necessary decoder software is stored in a memory hav- 
ing a low access speed, it is possible to decide the infor- 
mation required by a decoder and when the information 
is required by a timer or counter. In this case, tiie infor- 



mation for tiie priority of a timer or counter or the infor- 
mation for the priority of data processing is unnecessary 
for AL Information depending on the purpose. 
10126] Rgures 5(a) and 5(b) and Figures 6(a) to 6(d) 
are illustrations fbr explaining a method for adding AL 
information. 

[0127] By sending the conti'ol information for commu- 
nicating whether to add AL to only the head of the data 
to be transmitted as shown In Figure 5(a) or whether to 
add AL to each data piece after decomposing the data 
to be transmitted (ES) into one data piece or more to a 
receiving terminal as shown in Figure 5(b), it is possible 
to select the grading for handling transmission informa- 
tion. Adding AL to subdivk:led data Is effective when 
access delay is a problem. 

[0128] As described above, to previously communi- 
cate recombination of data control Information at tiie 
receiving skie or change of methods for arranging data 
control information to daUi to a receiving-side terminal, 
receiving-terminal conrespondence can be smootiily 
performed by using the expression of a flag, counter, or 
timer and thereby, preparing the expression as AL infor- 
mation or as a communication header to communicate 
It to the receiving terminal. 

[0129] In tiie case of tiie above examples, a method 
for avoiding duplication of the header of RTP (or com- 
munication header) with AL information and a method 
for extending the communication header of RTP or AL 
Information are described. However, It Is not always 
necessary for the present invention to use RTR For 
example, it is possible to newly define an original com- 
munication header or AL information by using UDP or 
TCP. Though the internet profile uses RTP sometimes, 
a multifunctional header such as RTP is not defined in 
tiie Raw profile. The following four types of concepts are 
oonsKlered for AL information and communication 
header (see Figures 6(a) to 6(d)). 

(1) The header information of RTP or AL informa- 
tion IS corrected and extended so that the header 
information already assigned to RTP and that 
already assigned to AL are not overlapped (particu- 
larly, the information for a time stamp is overlapped 
and tiie priority information for a timer, counter, or 
data processing becomes extension information). 
Or. it is possible to use a method of not extending 
tiie header of RTP or not consklering duplication of 
AL information with information of RTR They corre- 
spond to the contents having been shown so far. 
Because a part of RTP is already practically used 
fbr H.323. it Is effective to extend RTP having com- 
patibility (See Figure 6(a).) 

(2) Independently of RTP, a communication header 
is simplified (for example, using only a sequence 
number) and remainder is provided for AL informa- 
tion as multifunctional control Information. Moreo- 
ver, by making it possible to variably set Items used 
for AL information before communication, it Is pos- 
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sible to specify a flexible transmission format. (See 
Figure 6(b).) 

(3) IrKlependently of RIP. AL information is simpli- 
fied (for an extreme example, no information is 
added to AL] and every control information is pro- 
vided for a communication header. A sequence 
number, time stamp, marker bit. payload type, and 
object ID frequently used as communication head- 
ers are kept as fixed information and data-process- 
ing priority information and timer information are 
respectively provided witii an identifier showing 
whether extended information is present as 
extended information to refer to the extended infor- 
mation if the information is defined. (See Figure 
6(c).) 

(4) Independentiy of RTR a communication header 
and AL information are simplified and a format is 
defined as a packet separate from tiie communica- 
tion header or AL information to transmit the format. 
For ^cample, a mefliod is also considered in which 
only a marker bit, time stamp, and object ID are 
defined for AL information, only a sequence 
number is defined for a communication header, and 
payload information, data-processing priority infor- 
matfon, and timer information are defined as a 
transmission packet (second packet) separate from 
the above information and transmitted. (See Figure 
6(d).) 

[0130] As described above, when oonsklering a pur- 
pose and header information already added to a pfoture 
or audio, it is preferal^le so as to be able to freely define 
(customize) a packet (second packet) to be transmitted 
separately from a communication header, AL informa- 
tion, or data in accordance with the purpose. 
[0131] Figure 7 is an illustratfon for explaining a 
method for transmitting information by dynamically mul- 
tiplexing and separating a plurality of logical transmls- 
sion lines. The number of logical transmission lines can 
be decreased by providing an information multiplexing 
section capable of starting or ending multiplexing of the 
information for logical transmission lines for transmitting 
a plurality of pieces of data or control information in 
accordance witii the designation by a user or the 
number of logical transmission lines for a transmitting 
section and an information separating section for sepa- 
rating multiplexed information for a reception control 
section. 

[01 32] In Figure 7, the information multiplexing section 
Is referred to as "Group MUX" and specifically, It is pos- 
sible to use a multiplexing system such as H.223. It is 
possible to provide the Group MUX for a transmit- 
ting/receiving terminal. By providing the Group MUX for 
a relay router or terminal, It Is possible to correspond to 
a narrow-band communication channel. Moreover, by 
realizing Group MUX witii H.223, it is possible to inter- 
connect H.223 and H.324. 

[0133] To quickly fetch tiie control information (multi- 



plexing control information) for tiie information multi- 
plexing section. It is possible to reduce a delay due to 
multiplexing by transmitting the control information in 
tiie informatfon multiplexing section tiirough another 

5 logical transmission line witiiout multiplexing the control 
Information w'rth data by tiie information multiplexing 
section. Thereby, it is possit>le for a user to select 
whether to keep tiie consistency with conventional mul- 
tiplexing or reduce a delay due to multiplexing by com- 

10 munlcating and transmitting whetiier to multiplex the 
control information concerned with the information mul- 
tiplexing section with data and transmit tiiem or transmit 
tiie control information through another logical transmis- 
sion tine witiiout multiplexing tiie information witii the 

IS data. In tiiis case, tiie multiplexing control information 
concerned witii the information multiplexing section is 
information showing the content of multiplexing about 
how the information multiplexing section performs multi- 
plexing for each piece of data. 

20 [0134] As described above, similarly, it is possible to 
transmit tiie notification of a metiiod for transmitting at 
least the information for communicating tiie start and 
end of multiplexing, information for communicating the 
combination of logical transmission lines to be multi- 

25 ple)^, and control Informatfon concerned with multi- 
plexing (multiplexing control information) as control 
Information in accordance with an expression metiiod 
such as a flag, counter, or timer or reduce tiie setup time 
at tiie receiving side by transmitting data control infor- 

30 mation to a receiving-side tenninal togetiier with data. 
Moreover, as previously described, it is possible to pro- 
vide an item for expressing a flag, counter, or timer for 
tiie transmission header of RTF. 
[0135] When a plurality of information multiplexing 

35 sections or a plurality of Information separating sections 
are present, it is possible to kJentify to which information 
multiplexing section the control information (multiplex- 
ing control information) belongs by transmitting the con- 
trol information (multiplexing control information) 

40 togetiier with an identifier for identifying an information 
multiplexing section or information separating section. 
The control information (multiplexing control informa- 
tion) includes a multiplexing pattern. Moreover, by using 
a table of random number and thereby, deciding an 

45 identifier of an information multiplexing section or infor- 
mation separating section between temiinals, it is pos- 
sible to generate an identifier of tiie information 
multiplexing section. For example, it is possible to gen- 
erate random numbers in a range determined between 

50 transmitting and receiving terminals and use the largest 
value for the identifier (identification number) of the 
information multiplexing section. 
[0136] Because the data multiplexed by the informa- 
tion multiplexing section Is conventionally different from 

55 tiie media type defined in RTF, It Is necessary to define 
the information showing tiiat it is information multiplexed 
by tiie information multiplexing section (new media type 
H.223 is defined) for tiie payload type of RTF. 
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[01 37] By arranging the information to be transmitted 
by or recorded in the information multiplexing section in 
the sequence of control information and data informa- 
tion so as to improve the access speed to multiplexed 
data, It is expected to quickly analyze multiplexed infor- 5 
mation. Moreover, It is possible to quickly analyze 
header information by fixing an item which is described 
in accordance with the data control information added to 
control information and adding and multiplexing an 
Identifier (unique pattern) different from data. 10 
[0138J Figure 8 is an illustration for explaining tfie 
transmission procedure of a broadcasting program. By 
using the relation between the identifier of a logical 
transmission line and the Identifier of a broadcasting 
program as tiie infbmiation of the broadcasting program is 
and thereby, transmitting control information or adding 
the identifier of a broadcasting program to data as data 
control information (AL information), it is possible to 
identify tiiat the data transmitted through a plurality of 
transmission lines is broadcasted for which program. 20 
Moreover, by transmitting the relation between tiie iden- 
tifier of data (SSRC in tiie case of RTP) and tiie identi- 
fier of a logical transmission line (e.g. port number of 
LAN) to a receiving-side terminal as control information 
and transmitting con^espondlng data after it is confirmed 2s 
that the control information can be received by tiie 
receiving-side terminal (Ack/Reject), it is possible to 
form the correspondence between data pieces even if 
control information and data are respectively transmit- 
ted through an independent transmission line. 30 
[0139] By combining an identifier showing the trans- 
mission sequence of broadcasting programs or data 
pieces with the information for a counter or timer for 
showing a term of validity in which broadcasting pro- 
gram or data can be used as information, adding the 35 
combined Identifier and information to tiie broadcasting 
program or data, and transmitting them, it is possible to 
realize broadcasting without return channel (when the 
term of validity almost ©cpires, reproduction of the infor- 
mation or data for a broadcasting program is started 40 
even if information is Insufficient). Moreover, a method 
can be considered in which control information and data 
are broadcasted without being separated from each 
other by using ttie address of a single communication 
port (multicast address). 45 
[0140] In the case of communication with no back 
channel, it Is necessary to transmit control information 
sufficientiy before transmitting data so as to enable the 
receiving terminal to know a structural information of 
data. Moreover, control information should be transmit- so 
ted through a transmission channel free from packet 
loss and having a high reliability. However, when using a 
transmission channel having a low reliability, it is neces- 
sary to cyclically transmit the control information having 
the same transmission sequence number. This is not ss 
restricted to the case of transmitting the control informa- 
tion concerned witii a setup time. 
[0141 ] Moreover, it is possible to flexibly control and 



transmit data by selecting an item which can be added 
as data control information (e.g. access flag, random 
access flag, data reproducing time (PTS). or data- 
processing-priority information), deciding whether to 
transmit the data control informatfon together witii tiie 
identifier (SSRC) of data as control information tiirough 
a logical transmission line different from tiiat of tiie data 
or transmit the data control Information as data control 
information (information for AL) together with tiie data at 
tiie transmitting skle before transmitting the data, and 
communicating and transmitting tiie data to the receiv- 
ing side as confrol information. 
[01 42] Thereby, it is possible to ta'ansmit data informa- 
tion without adding Information to AL Therefore, to 
transmit tiie data for a pfoture or audio by using RTP, it 
is unnecessary to extend the definition of tiie payload 
having been defined so far. 

[0143] Figures 9(a) and 9(b) are illustrations showing 
a picture or audio transmission metiiod conskJering the 
read time and rise time of program or data Particularly, 
when the resources of a terminal are limited like the 
case of satellite broadcasting or a portable terminal 
having no return channel and being unidirectional, pro- 
gram or data is present and used at a receiving-skie ter- 
minal, a necessary program (e.g. H.263, MPEG1/2, or 
software of audio decoder) or data (e.g. video data or 
audio data) is present in a memory (e.g. DVD, hard disk, 
or file server on network) requiring a lot of read time, it 
is possible to reduce the setup time of program or data 
required in advance by previously receiving it as control 
information or receiving it together with data as data 
control information in accordance with the expression 
metiiod such as the identifier for identifying the program 
or data, identifier (e.g. SSRC. or Logical Channel 
Number) of a stream to be transmitted, or a flag, counter 
(count-up/down), or timer for estimating the point of time 
necessary for a receiving terminal (Figure 18). 
[01 44] When program or data is transmitted, by trans- 
mitting tiie program or data from the transmitting side 
togetiier witii tiie information showing the storage desti- 
nation (e.g. hard disk or memory) of the program or data 
at a receiving terminal, time required for start or read, 
relation t^etween tiie type or storage destination of a ter- 
minal and the time required for start or read (e.g. rela- 
tion between CPU power, storage device, and average 
response time), and utilization sequence, it is possible 
to schedule the storage destination and read time of the 
program or data if tiie program or data necessary for the 
receiving terminal is actually required. 
[0145] Figures 10(a) and 10(b) are illustrations for 
explaining a method for corresponding to zapping 
(channel change of TV). 

[0146] When it is necessary to execute a program at 
a receiving terminal differentiy from the case of conven- 
tional satellite broadcasting for receiving only pictures, 
tiie setup time until the program is read and started is a 
large problem. The same is true for tiie case in which 
available resources are limited like tiie case of a porta- 



15 



29 



EP0905 976A1 



30 



ble terminal. 

[0147] It is expected that the setup time at a receiving- 
side terminal can be decreased by (a) using a main 
iooking-llstening section by which the user looks at and 
listens to. and an auxiliary looking-listening section in 
which a receiving terminal cyciicaily monitors programs 
other than the program looked and listened by a user 
and receiving the relation between identifier lor identify- 
ing program or data required in advance, information for 
a flag, counter, or timer for estimating the point of time 
necessary for the receiving terminal, and program as 
control information (information transmitted by a packet 
different from that of data to control terminal processing) 
or as data control information (information for AL), and 
preparing read of tiie program or data together witii data 
as one of tiie settlement measures when to program or 
data necessary for a program otiier tiian tiie program 
looked and listened by the user is present in a memory 
requiring a lot of time for read. 
[0148] It Is possible to prevent a screen from stopping 
under setup by setting a broadcasting channel for 
broadcasting only heading pictures of tiie pictures 
broadcasted tiirough a plurality of channels and switch- 
ing programs by a user, and thereby, when necessary 
program or data is present in a memory requiring a lot 
of time for read, temporarily selecting the heading pic- 
ture of a program required by the user and showing it for 
the user or showing that program or data is currently 
read, and restarting tiie program required by tiie user 
after necessary program or data is read by tiie memory 
as tiie second one of the settlement measures. The 
above heading pictures include broadcasted pictures 
obtained by cyclically sampling programs broadcasted 
through a plurality of channels. 
[0149] Moreover, a timer is a time expression and 
shows the point of time when a program necessary to 
decode a data stream sent from the transmitting side is 
necessary A counter is the basic time unit determined 
between transmitting and receiving terminals, which 
can be information showing what-tii time. A flag is trans- 
mitted and communicated together with the data trans- 
mitted before the time necessary for setup or corrtrol 
information (information transmitted tiirough a packet 
different from tiiat of data to control terminal process- 
ing). It is possible to ti-ansmit tiie timer and counter by 
embedding ttiem in data or transmit them as control 
information. 

[0150] Furtiiermore. to decide a setup time, the time 
in which setup is performed can be estimated by, when 
using a transmission line such as ISDN operating on the 
clock base, using a transmission serial number for iden- 
tifying a transmission sequence as transmission control 
information in order to communicate from the transmit- 
ting terminal to the receiving terminal a time point when 
program or data is required and thereby communicating 
the serial number to a receiving terminal together with 
data as data control information or as control informa- 
tion. Furthermore, when a transmission time is fluctu- 



ated due to jitter or delay like internet, it is necessary to 
add the transmission time to the setup time by consider- 
ing the propagation delay of transmission in accordance 
viritii jitter or delay time by tiie means for realizing RTCP 
5 (media transmission protocol used for internet). 

[01 51 ] Figures 11 (a) to 1 9(b) are illustrations showing 
specific examples of protocols actually transferred 
between terminals. 

[01 52] A transmission format and a transmission pro- 

10 cedure are described in ASN.1 . Moreover, tiie transmis- 
sion format is extended on the basis of H.245 of ITU. As 
shown in Figure 11(a), objects of a picture and audio 
can have a hierarchical structure. In the case of this 
example, each object ID has tiie attributes of a broad- 

15 casting-program identifier (program ID) and an object ID 
(S SRC) and the structural information and synthesizing 
metiiod between pictures are described by a script lan- 
guage such as Java or VRML 
[0153] Figure 11(a) is an illustration showing exam- 

20 pies of ttie relation between objects. 

[01 54] In Figure 1 1 (a), objects are media such as an 
audio-video, CG, and text. In tiie case of the examples 
in Figure 11(a), objects constitute a hierarchical struc- 
ture. Each object has a program number "Program ID" 

26 corresponding to TV channel) and an object identifier 
"Object ID" for identifying an object. When transmitting 
each object in accordance with RTF (media transmis- 
sion protocol for transmitting media used for internet, 
Realtime Transfer Protocol), it is possible to easily iden- 

30 tify tiie object by making tiie object identifier correspond 
to SSRC (synchronous source identifier). Moreover, it is 
possible to describe the structure between objects with 
a description language such as JAVA or VRML 
[0155] Two types of methods for transmitting the 

35 objects are considered. One is the broadcasting type in 
which the objecte are unilaterally transmitted from a 
transmitting-side terminal. The other is the type (com- 
munication type) for transferring the objects between 
fransmitting and receiving terminals (terminals A and 

40 B). 

[Q156] For example, it is possible to use RTP as a 
transmission metiiod in the case of internet. Control 
information is transmitted by using a transmission chan- 
nel referred to as LCNO in tiie case of the standard for 

45 video telephones. In tiie case of tiie example in Figure 
1 1(a), a plurality of transmission channels are used for 
transmission. The same program channel (program ID) 
is assigned to these channels. 
[01 57] Figure 1 1 (b) is an illustration for explaining how 

50 to realize a protocol for realizing tiie functions described 
for the present invention. The transmission protocol 
(H.245) used for tiie video-telephone standards (H.324 
and H.323) is described below. The functions described 
for the present invention are realized by extending 

55 H.245. 

[01 58] The description method shown by the example 
in Rgiu-e 11(b) is the protocol description method 
referred to as ASN.1. "Terminal Capability Set" 
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expresses the performance of a terminal. In the case of 
the example in Figure 11(b), the function described as 
**mpeg4 Capability" is extended for the conventional 
H.245. 

[0159] In Figure 12, "mpeg4 Capability" describes the 
maximum number of pictures "Max Number Of pictures" 
and the maximum number of audio ("Max Number Of 
Audio") which can be simultaneously processed by a 
terminal and the maximum nunflber of multiplexing func- 
tions fMax Number Of Mux") which can be realized by 
a terminal. 

[01 60] In Figure 1 2. these are expressed as the max- 
imum number of objects ("Number Of Process Object") 
which can be processed. Moreover, a flag showing 
whether a communication header (expressed as AL in 
Figure 12) can be changed is described. When the 
value of the flag Is true, the communication header can 
be changed. To communicate the number of objects 
which can be processed between terminals to each 
other by using "MPEG4 Capability", the communicated 
side returns "MEPG4 Capability Ack" to a terminal from 
which "MEPG4 Capability" is transmitted if the commu- 
nicated side can accept (process) the objects but 
returns "MEPGi4 Capability Reject" to the terminal if not 
[0161] Rgure 13(a) shows how to describe a protocol 
for using the above Group MUX for multiplexing a plural- 
ity of logical channels to one transmission channel 
(transmission channel of LAN in the case of this exam- 
ple) in order to share the transmission channel by logi- 
cal channels. In the case of the example in Rgure 13(a), 
multiplexing means (Group MUX) is made to corre- 
spond to the transmission channel ("LAN Port Number") 
of LAN (Local Area Network). "Group Mux ID" Is an 
Identifier for identifying the multiplexing means. To 
share the multiplexing means by terminals by using 
"Create Group Mux" and perform comnfiunication 
between the terminals, the communicated side returns 
"Create Group Mux Ack" to a terminal from which "Cre- 
ate Group Mux" is transmitted if the side can accept 
(use) the multiplexing means but returns "Create Group 
Mux Reject" to the terminal if not. Separating means 
serving as means for performing an operation reverse to 
that of the multiplexing means can be realized by the 
same method. 

[01 62] in Figure 1 3(b), a case of deleting already-gen- 
erated multiplexing means Is described. 
[0163] In Figure 13(c), the relation between the trans- 
mission channel of LAN and a plurality of logical chan- 
nels is described. 

[0164] The transmission channel of LAN is described 
in accordance with "LAN Port Number" and the logical 
channels are described in accordance with "Logical 
Port Number". 

[0165] In the case of the examples in Figure 13(c). it 
is possible to make the transmission channel of one 
LAN correspond to up to 15 logical channels. 
[0166] In Figure 13. when the number of MUXs that 
can be used is only one, Group Mux ID is unnecessary 



Moreover, to use a plurality of Muxes, Group Mux ID Is 
necessary for each command of H.223. Furthermore, it 
is possible to use a flag for communicating the relation 
between ports used between tiie multiplexing means 

5 and separating means. Furtiiermore, it is possible to 
use a command making it possible to select whether to 
multiplex control information or transmit the Information 
tiirough another logical transmission line. 
[01 67] In tiie case of the explanation in Figures 13(a) 

10 to 13(c), the transmission channel uses LAN. However, 
it is also possitile to use a system using no internet pro- 
tocol like H.223 or MPEG2. 

[0168] In Figure 14, "Open Logical Channel" shows 
tiie protocol desaiption for defining the attribute of a 

IS transmission channel. In tiie case of the example in Rg- 
ure 14, "MPEG4 Logical Channel Parameters" is 
extended and defined for tiie protocol of H.245. 
[01 69] Figure 1 5 shows tiiat a program number (cor- 
responding to a TV channel) and a program name are 

20 made to oonrespond to tiie transmission channel of LAN 
("MPEG4 Logical Channel Parameters"). 
[0170] Moreover, in Figure 15, "Broadcast Channel 
Program" denotes a description method for transmitting 
the correspondence between LAN transmission chan- 

25 nel and program number in accordance witii tiie broad- 
casting type. The example In Figure 15 makes it 
possible to transmit the con^espondence between up to 
1,023 transmission channels and program numbers. 
Because transmission is unilaterally performed from tiie 

30 transmitting side to the receiving side in the case of 
broadcasting, it is necessary to cyclically transmit these 
pieces of information t>y considering the loss during 
transmission. 

[0171] In Figure 16(a). the attribute of an object (e.g. 

35 picture or audio) to be transmitted as a program is 
described ("MPEG4 Object Classdelinrtion"). Object 
information ("Object Structure Element") is made to cor- 
respond to a program identifier ("Program ID"). It is pos- 
sible to make up to 1,023 objects correspond to 

40 program identifiers. As the object information, a LAN 
transmission channel ("LAN Port Number"), a flag 
showing whetiier scramble is used ("Scramble Flag"), a 
field for defining an offset value for changing the 
processing priority when a terminal Is overloaded 

45 fCGD Offset), and an identifier (Media Type) for identi- 
fying a type of the media (picture or audio) to be trans- 
mitted are described. 

[01 72] In the case of the example in Figure 1 6(b), AL 
(in this case, defined as additional information neces- 
50 sary to decode pictures for one frame) is added to con- 
trol decoding of ES (in this case, defined as a data string 
corresponding to pictures for one frame). As AL infor- 
mation, the following are defined. 

55 (1) Random Access Flag (flag showing whether to 
be Independently reproducible, true for an intra- 
frame encoded picture frame) 
(2) Presentation Time stamp (time displayed by 
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frame) 

(3) CGD Priority (Value of priority for deciding 
processing priority when terminal is overloaded) 

[0173] The example shows a case of transmitting the 
data string for one frame by using RTP (protocol lor 
transmitting continuous media through internet, Real- 
time Transfer Protocol). "AL Reconfiguration" Is a trans- 
mission expression for changing the maximum value 
that can be expressed by the above AL 
[0174] The example in Figure 1 6(b) makes it possible 
to express up to 2 bits as "Random Access Flag Max 
Bit". For example, when there Is no bit, Random Access 
Flag is not used. When there are two bits, the maximum 
value is equal to 3. 

[0175] Moreover, the expression with a real number 
part and a mantissa part is allowed (e.g. 3^6). When no 
data is set, an operation under the state decided by 
default is allowed. 

[0176] In Rgure 17. "Setup Request" shows a trans- 
mission expression for transmitting a setup time. "Setup 
Request" is transmitted before a program is transmitted, 
a transmission channel number ("Logical Channel 
Number") to be transmitted, a program ID ("execute 
Program Number) to be executed, a data ID ("data 
Number") to be used, and the ID of a comnfiand ("exe- 
cute Command Number") to be executed are made to 
con'espond to each other and transmitted to a receiving 
terminal. Moreover, an execution authorizing flag 
(llag"). a counter ("counter") describing whether to start 
execution when receiving Setup Request how many 
times, and a timer value ("timer") showing whether to 
start execution after how many hours pass can be used 
as other expression methods by making them corre- 
spond to transmission channel numbers. 
[0177] Rewriting of AL information and securing of 
rise time of Group Mux are listed as examples of 
requests to be demanded. 

[0178] Figure 18 is an illustration for explaining a 
transmission expression for communicating whether to 
use the AL described for Figure 16(b) from a transmit- 
ting terminal to a receiving terminal ("Control AL defini- 
tion"). 

[0179] In Figure 18, if "Random Access Flag Use" is 
true. Random Access Flag is used. If not. it is not used. 
It Is possible to transmit the AL change notification as 
control information through a transmission channel sep- 
arate from that of data or transmit it through the trans- 
mission channel same as that of data together with the 
data. 

[0180] A decoder program is listed as a program to be 
executed. Moreover, a setup request can be used for 
broadcasting and communication. Furthermore, which 
item serving as control information is used as Al infor- 
mation is designated to a receiving terminal in accord- 
ance with the above request. Furthermore, it is possible 
to designate which item is used as communication 
header, which item is used as AL information and which 



item is used as control informatfon to a receiving termi- 
nal. 

[01 81 ] Figure 1 9(a) shows the exanple of a transmis- 
sion expression for changing the structure of header 

5 information (data control information, transmission con- 
trol information, and control inforniation) to be transmit- 
ted by using an information frame identifier ("header 
ID") between transmitting and receiving terminals in 
accordance with the purpose. 

10 [0182] In Rgure 19(a), "class ES header" separates 
the structure of the data control information to be trans- 
mitted through a transmission channel same as that of 
data firom that of the information with which transmis- 
sion control informatfon Is transmitted between trans- 

15 mrtting and receiving terminals in accordance with an 
information frame identifier. 

[0183] For example, only the item of "buffer Size ES" 
is used when the value of "header ID" Is 0 but the item 
of "reserved" is added when the value of "header ID" is 
20 1. 

[0184] Moreover, by using a default identifier ("use 
Header Extension"), rt is decided whetiier to use a 
default-type information frame. When "use Header 
Extension" is true, an item in an if-statement is used. It 

25 Is assumed that these pieces of structural information 
are previously decided between transmitting and receiv- 
ing terminals. Furthermore, it is possible to use a struc- 
ture for using either of an informatfon frame identifier 
and a default identifier. 

30 [0185] In Figure 19(b), "AL configuration" shows an 
example for changing tiie structure of control informa- 
tion to be transmitted through a transmission channel 
different from that of data between transmitting and 
receiving terminals in accordance witii the purpose. The 

35 usage of an information frame identifier and that of a 
default identifier are tiie same as the case of Figure 
19(a). 

[0186] In tiie case of the present invention, methods 
for realizing a system for simultaneously syrrthesizing 
40 and displaying a plurality of pictures and a plurality of 
audio are specifically described from tiie following view- 
points. 

(1) A method for transmitting (communicating and 
45 broadcasting) a picture and an audio through a plu- 
rality of logical transmission lines and controlling 
tiiem. Particularly, a method for respectively trans- 
mitting control information and data through an 
independent logical transmission line is described. 
so (2) A method for dynamically changing header 
information (AL information) added to the data for a 
picture or audio to be transmitted. 
(3) A method for dynamically changing communica- 
tion header information added for transmission. 
55 Specifically, for Items (2) and (3), a metiiod for 

uniting and controlling the information overlapped 
In AL information and communication header and a 
method for transmitting AL information as control 
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information are described. 

(4) A method for dynamically multiplexing and sep- 
arating a plurality of logical transmission lines and 
transmitting information. 

A method for economizing the number of chan- s 
neis of transmission lines and a method for realiz- 
ing efficient muitipl^'ng are described. 

(5) A method for reading a program or data and 
transmitting pictures and audio considering a rise 
time. Moreover, a method for reducing an apparent 
setup time for various functions and purposes is 
described. 

(6) A method for transmitting a picture or audio for 
zapping. 

[0187] The present invention is not restricted to only 
synthesis of two-dimensional pictures. It is also possible 
to use an expression method of combining a two-dimen- 
sional picture with a three-dimensional picture or 
include a picture synthesizing method for synthesizing a 
plurality of pictures so that they are adjacent to each 
other like a wide-visual-field picture (panoramic picture). 
[0188] Moreover, the present invention does not pur- 
pose only such communication systems as bidirectional 
CATV and B-ISDN. I=br example, it is possible to use 
radio waves (e.g. VHP band or UHF band) or a broad- 
casting satellite for transmission of pictures and audio 
from a center-side terminal to a home-side terminal and 
an analog telephone line or N-ISDN for transmission of 
information from a home-side terminal to a center-side 
terminal (it is not always necessary that pictures, audia 
and date are multiplexed). 

[0189] Moreover, it is possible to use a communication 
system using radio such as IrDA, PHS (Personal Handy 
Phone), or radio LAN. Furthermore, a purposed termi- 
nal can be a portable terminal such as a portable infor- 
mation terminal or a desktop terminal such as a setup 
BOX or personal computer. Furthermore, a video tele- 
phone, multipoint monitoring system, multimedia data- 
base retrieval system, and game are listed as 
application fields. The present invention Includes not 
only a receiving terminal but also a server and a 
repeater to be connected to a receiving terminal. 
[0190] Furthermore, in the case of the above exam- 
ples, a method for avoiding the overlap of the (commu- 
nication) header of RTP with AL information and a 
method for extending the communication header of RTP 
or AL information are described. However, it is not 
always necessary for the present invention to use RTP. 
For example, it is also possible to newly define an origi- 
nal communication header or AL information by using 
UDP or TOR Though an internet profile uses RTP 
sometimes, a multifunctional header such as RTP is not 
defined for a Raw profile. There are four types of con- 
cepts about AL information and communication header 
as described above. 

[01 91 ] Thus, by dynamically deciding tiie information 
frame of data control information, transmission control 



information, or control information used by the transmit- 
ting and receiving terminals (e.g. information frame 
including the sequence of information to be added and 
the number of bits for firstly assigning a random access 
flag as 1-bit flag infonnation and secondly assigning 16 
bits in the form of a sequence number), it is possible to 
change only an information frame con^esponding to tiie 
situation in accordance witii tiie purpose or transmis- 
sion line. 

[01 92] The frame of each piece of information can be 
any one of the frames already shown in Figures 6(a) to 
6(d) and in the case of RTP, the data control information 
(AL) can be the header information for each medium 
(e.g. in tiie case of H.263. tiie header information of tiie 
vkieo or that of tiie payload intrinsic to H.263), transmis- 
sion control information can be tiie header information 
of RTP, and control information can be the information 
for controlling RTP such as RTCR 
[01 93] Moreover, In the case of a publicly-known infor- 
mation frame previously set between transmitting and 
receiving terminals, by providing a default identifier for 
showing whether to process information by transmitting 
and receiving for data control information, fransmission 
control information, and control information (information 
tiransmitted tiirough a packet different from tiiat of data 
to control terminal processing) respectively, it is possi- 
ble to know whether information frames are changed. 
By setting tiie default identifier and communicating the 
changed content (such as change of time stamp infor- 
mation from 32 to 16 bits) only when change is per- 
formed in accordance with the metiiod shown in Figure 
1 6, rt is prevented to transmit unnecessary configuration 
information even when frame information of information 
is not changed. 

[0194] For example, the folfowing two methods are 
considered to change information frames of data control 
Information. Rrst, to describe a method for changing 
information frames of data control information in data, 
the default identifier (to be written in a fixed region or 
position) of the information present in the data 
described for the information frame of data control infor- 
mation is set and then, information frame change con- 
tents are described. 

[0195] To change information frames of data control 
information by describing a method for changing only 
the information frames of data in the control information 
(information frame control information) as another 
metiiod, a default identifier provided for control informa- 
tion is set, the contents of the information frames of the 
data control information to be changed are described, 
and it is communicated to a receiving terminal in 
accordance wifli ACK/Reject and confirmed tiiat tiie 
information frames of the data control information are 
changed and tiiereafter, the data in which information 
frames are changed is transmitted. Information frames 
of ti-ansmission conti-ol information and conti-ol informa- 
tion can be also changed in accordance with the above 
two methods (Rgure 19). 



15 



20 



25 



30 



35 



40 



45 



50 



19 



37 



EP0 905 976A1 



38 



[0196] More specifically, though the header infornia- 
tlon of MPEG2 is fixed, by providing a default identifier 
for a program map table (defined by PSI) for relating the 
video stream of MPEG2-Ts (transport stream) with the 
audio stream of rt and defining a configuration stream in 5 
which a method for changing frames of the information 
for the video stream and audio stream is described, it is 
possible to first interpret the configuration stream and 
then, interpret the headers of the video and audio 
streams in accordance with the content of the configura- 
tion stream when the defeult identifier is set It is possi- 
ble for the configurat'on stream to have the contents 
shown in Figure 19. 

[0197] The contents (transmitted-format information) 
of the present invention about a transmission method 
and/or a structure of the data to be transmitted corre- 
spond to, for example, an information frame in the case 
of the above embodiment. 

[0198] Moreover, for the above embodiments, a case 
of transmitting the contents to be changed concerned 
with a transmission method and/or the structure of the 
data to be transmitted is mainly described. However, it is 
also possible to use a structure for transmitting only the 
identifier for the contents. In this case, as shown in Fig- 
ure 44, it is also possible to use an audio-video transmit- 
ter provided with (1) transmitting means 5001 for 
transmitting the content concerned with a transmission 
method and/or the structure of the data to be transmit- 
ted or an identifier showing the content as the transmit- 
ted-format information through the transrrtission line 
same as that of the data to be transmitted or a transmis- 
sion line different from the former transmission line and 
(2) storing means 5002 for storing a plurality of types of 
the contents concerned with the transmission method 
and/or the structure of the data to be transmitted and a 
plurality of types of identifiers for the contents, in which 
the identifiers are included in at least one of the data 
control information, transmission control information, 
and information for controlling terminal-side processing. 
Moreover, as shown in Figure 45, it is possible to use an 
audio-video receiver provided with receiving means 
5101 for receiving the transmission format information 
transmitted from the audio-video transmitter and trans- 
mission information interpreting means 5102 for inter- 
preting the received transmission format information. 
Furthermore, the audio-video receiver can be consti- 
tuted with storing means 5103 for storing a plurality of 
types of contents concerned with the transmission 
method and/or the structure of the data to be transmit- 
ted and a plurality of types of identifiers for the contents 
to use the contents stored in the storing means to inter- 
pret the contents of the identifiers when receiving the 
identifiers as the transmission format information. 
[0199] More specifically, by preparing a plurality of 
types of information frames previously determined 
between transmitting and receiving terminals and trans- 
mitting identifiers for the above information frames and 
information frame identifiers for a plurality of types of 



data control information, a plurality of types of transmis- 
sion control information, and a plurality of types of con- 
trol information (information-frame control information) 
together with data or as control infonnation, it is possi- 
ble to identify a plurality of types of data control informa- 
tion, a plurality of types of transmission cont'd 
information, and a plurality of types of control informa- 
tion and optionally select the information frame of each 
type of information in accordance with the type of a 
medium to be transmitted or the size of a transmission 
line. Identifiers of the present invention correspond to 
tiie above information frame identifiers. 
[0200] It is possible to read and interpret these infor- 
mation identifiers and default identifiers even if informa- 
tion frames are changed at a receiving-side terminal by 
adding the identifiers to a predetermined fixed-lengtii 
region or predetermined position of the information to 
be transmitted. 

[0201] Moreover, in addition to tiie structures 
described for the above embodiments, it is possible to 
use a structure for temporarily selecting the caption pic- 
ture of a program to be looked and listened by tiie user 
and showing it for the user when it takes a lot of time to 
set up a necessary program or data by using a broad- 
casting channel for broadcasting only the heading pic- 
tures of pictures broadcasted through a plurality of 
channels and switching programs to be looked and lis- 
tened by the user. 

[0202] As described above, tiie present invention 
mates it possible to change frames of tiie information 
correspondlr)g to tiie situation in accordance with tiie 
purpose or transmission line by dynamically determin- 
ing the frame of data control information, fransmission 
control information, or control information used by trans- 
mitting and receiving terminals. 
[0203] Moreover, it is possible to know whether infor- 
mation frames are changed by providing a default kien- 
tifier for showing whether to transit or receive and 
process information by a puWicIy-known information 
frame previously set between transmitting and receiving 
terminals for data control information, transmission con- 
t'd information, and contrd information respectively 
and it is possible to prevent unnecessary configuration 
information from being transmitted even if information 
frames of information are not changed by setting a 
default identifier and communicating changed contents 
only when change is performed. 
[0204] Furthermore, it is possible to identify a plurality 
of types of data control information, a plurality of types 
of transmission control information, and a plurality of 
types of control information by preparing a plurality of 
information frames previously determined between 
ti*ansmitting and receiving terminals and transmitting 
information frame identifiers for identifying a plurality of 
types of data control information, a plurality of types of 
transmission control information, and a plurality of types 
of control information togetiier with data or as conf ol 
information and optionally select the information frame 
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of each type of information in accordance with the type 
of a medium to be transmitted or the size of a transmis- 
sion line. 

[0205] It is possible to read and interpret these infor- 
mation Identifiers and defoult identifiers even if infbrma- s 
tion frames are changed at a receiving-side terminal by 
adding the identifiers to a predetermined fixed-length 
region or predetermined position of tfie Information to 
be transmitted. 

[0206] Embodiments of tiie present invention are io 
described below by referring to tiie accompanying draw- 
ings. 

[0207] In this case, any one of tiie above-described 
problems (B1) to (B3) is solved. 

[0208] A "picture ( or video)" used for the present is 
Invention includes botii a static picture and a moving 
picture. Moreover, a purposed picture can be a two- 
dimensional picture such as a computer graphics (CG) 
or three-dimensional picture data constituted witii a 
wire-frame model. 20 
[0209] Figure 25 is a schematic block diagram of tiie 
picture encoder and a picture decoder of an embodi- 
ment of the present invention. 
[021 0] A transmission control section 401 1 for trans- 
mitting or recording various pieces of encoded infbrma- 25 
tion is means for transmitting tiie information for coaxial 
cable, CATV, LAN, or modem. A picture encoder 4101 
has a picture encoding section 4012 for encoding pic- 
ture information such as 1-1.263, MPEG1/2, JPEG, or 
Huffman encoding and the transmission control section 30 
4011. Moreover, a picture decoder 4102 has an output 
section 4016 constituted with a reception control section 
4013 for receiving various pieces of encoded informa- 
tion, a picture decoding section 4014 for decoding vari- 
ous pieces of received picture information, a picture 35 
syntiiesizing section 4015 for syntiiesizing one decoded 
picture or more, and an output section 4016 constituted 
with a display and a printer for oulputting pictures. 
[021 1 ] Figure 26 is a schematic block diagram of the 
audio encoder and an audio decoder of an emtsodiment 40 
of tiie present invention. 

[0212] An audio encoder(sound encorder) 4201 is 
constituted with a transmission control section 4021 for 
transmitting or recording various pieces of encoded 
information and an audio encoding section 4022 for 4S 
encoding such audio information such as G.721 or 
MPEG1 audio. Moreover, an audio decoder(a sound 
decoder) 4202 is constituted witii a reception control 
section 4023 for receiving various pieces of encoded 
information, an audio decoding section 4024 for decod- so 
ing the above pieces of audio information, an audio syn- 
tiiesizing section (a sound synthesizing section)4025 
for synthesizing one decoded audio or more, and output 
means 4026 for outputting audio. 

[021 3] Time-series data for audio or picture is specif i- ss 
cally encoded or decoded by the above encoder or 
decoder. 

[0214] The communication environments in Figures 



25 and 26 can be a communication environment in 
which a plurality of logical transmission lines can be 
used without considering multiplexing means like the 
case of internet or a communication environment in 
which multiplexing means must be considered like the 
case of an analog telephone or satellite broadcasting. 
Moreover, a system for bilaterally transferring a picture 
or audio between terminals like a video telephone or 
video conference or a system for broadcasting a broad- 
casting-type picture or audio on satellite broadcasting. 
CATV, or Internet is listed as a terminal connection sys- 
tem. 

[0215] Moreover, a method for syntiiesizing a picture 
and audio can be defined by describing a picture and an 
audio, structural infonmation for a picture and an audio 
(display position and display time), an audio-video 
grouping method, a picture display layer (deptii). and an 
object ID (ID for identifying each object such as a pic- 
ture or audio) and the relation between the attributes of 
tiiem witii a script language such as JAVA, VRML, or 
MHEG. A script describing a syntiiesizing metfiod is 
obtained from a network or local memory. 
[021 6] Moreover, it is possible to constitute a transmit- 
ting or receiving terminal by optionally combining an 
optional number of picture encoders, picture decoders, 
audio encoders, and audio decoders. 
[0217] Figure 27(a) is an illustration for explaining a 
priority adding section and a priority deciding section for 
controlling the priority for processing under overioad. A 
priority adding section 31 for deciding the priority for 
processing encoded information under overload In 
accordance witii a predetermined criteria by an encod- 
ing method such as H.263 or G.723 and relating the 
encoded information to the decided priority is provided 
for tiie picture encoder 4101 and audio encoder 4201. 
[0218] The criteria for adding a priority are scene 
change in the case of a picture and audio and audioless 
blocks in the case of a picture frame, stream, or audio 
designated by an editor or user. 
[021 9] A method for adding a priority to a communica- 
tion header and a metiiod for emk>edding a priority in the 
header of a bit stream to be encoded of a video or audio 
under encoding are considered as priority adding meth- 
ods for defining a priority under overioad. The former 
metiiod makes it possible to obtain the information con- 
cerned with & priority without decoding the information 
and the latter method makes it possible to independ- 
ently handle a single bit stream witiiout depending on a 
system. 

[0220] As shown in Figure 27(b), when priority infor- 
mation is added to a communication header and one 
picture frame (e.g. intira-frame encoded 1-frame or inter- 
frame encoded P- or B-frame) is divided into a plurality 
of transmission packets, a priority is added only to a 
communication header for transmitting the head of a 
picture frame accessible as single information in the 
case of a picture (when priorities are equal in the same 
picture frame, it is possible to assume that the priorities 
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are not changed until the head of the next accessible 
picture frame appears). 

[0221] Moreover, in the case of a decoder, a priority 
deciding section 32 for deciding a processing method is 
provided for the picture decoder 4102 and audio s 
decoder 4202 in accordance with the priorities of vari- 
ous pieces of encoded information received under over- 
load. 

[0222] Figures 28(a) to 28(c) are Illustrations for 
explaining the grading for adding a priority. Decoding is io 
performed by using two types of priorities for deciding 
the priority for processing under overload at a terminal. 
[0223] That is, a stream priority (Stream Priority; inter- 
time-series-data priority) for defining the priority for 
processing under overload in bit streams such as pic- is 
ture and audio and a frame priority (Frame Priority; 
intra-time-series-data priority) for defining the priority for 
processing under overload in frames such as picture 
frames in the same stream are defined (see Figure 
28(a)). 20 
[0224] The former stream prtority makes it possible to 
handle a plurality of videos or audios. The latter frame 
priority mal<es it possible to add a different priority to a 
picture scene change or the same intra-f rame encoded 
picture frame (1-frame) in accordance with the intention 25 
of an editor. 

[0225] A value expressed by the stream priority repre- 
sents a case of handling it as a relative value and a case 
of handling it as an absolute value (see Figures 28(b) 
and 28(c)). 30 
[0226] The stream and frame priorities are handled by 
a repeating terminal such as a router or gateway on a 
network and by transmitting and receiving terminals in 
the case of a terminal. 

[0227] Two types of methods for expressing an abso- 3S 
lute value or relative value are considered. One of them 
is the method shown in Rgure 28(b) and the other of 
them is the method shown in Figure 28(c). 
[0228] In Figure 28(b), the priority of an absolute value 
is a value showing the sequence in which picture 40 
streams (video streams) or audio streams added by an 
editor or mechanically added are processed (or to be 
processed) under overload (but not a value considering 
the load fluctuation of an actual network or terminal). 
The priority of a relative value is a value for changing the 4S 
value of an absolute priority in accordance with the load 
of a terminal or network. 

[0229] By dividing a priority into a relative value and 
an absolute value to control the values and thereby 
changing only relative values at the transmitting side or so 
by a repeater in accordance with the load fluctuation of 
a network or the like, it is possible to record the value of 
an absolute value into a hard disk or VTR while leaving 
the absolute priority added to a video or audio stream. 
Thus, when the value of the absolute priority is ss 
recorded, it is possible to reproduce a picture or audio 
that Is not influenced by the load fluctuation of a network 
or the Wke. Moreover, it is possible to transmit a relative 



or absolute priority through a control channel independ- 
ently of data. 

[0230] Moreover, in Rgure 28(b), it is possible to fine 
the grading compared to a stream priority and handle a 
fi^me priority for defining the priority for frame process- 
ing under overload as the value of a relative priority or 
handle it as the value of an absolute priority. For exam- 
ple, by describing an absolute frame priority in encoded 
picture information and describing a relative frame prior- 
ity conresponding to the atssolute priority added to tiie 
picture frame in the communication header of a commu- 
nication packet for transmitting encoded information in 
order to reflect the load fluctuation of a network or termi- 
nal, it is possible to add a priority corresponding to tiie 
load of a network or terminal even at a frame level while 
leaving an original priority. 

[0231] Moreover, it is possible to transmit a relative 
priority by describing tiie relation with a frame not in a 
communication header but in a control channel inde- 
pendently of data. Thereby, It is possible to record dala 
into a hard disk or VTR while leaving an absolute priority 
originally added to a picture or audio stream. 
[0232] Furthermore, in Figure 28(b), when reproduc- 
ing data at a receiving terminal while transmitting the 
data through a network without recording the data at the 
receiving terminal, it is possible to compute tiie value of 
an absolute priority and tiiat of a relative priority at 
frame and stream levels at the transmitting side and 
tiiereafter transmit only absolute values because it is 
unnecessary to control absolute and relative values by 
separating tiiem from each other at a receiving terminal. 
[0233] In Figure 28(c), the priority of an absolute value 
is a value uniquely determined between frames 
obtained from tiie relation between Stream Priority and 
Frame Priority The priority of a relative value is a value 
showing the sequence in which picture stireams or audio 
streams added by an editor or mechanically added are 
processed (or to be processed) under overload. In tiie 
case of tiie example in Figure 28(c), tiie frame priority of 
a picture or audio stream (relative; relative value) and 
tiie stream priority for each stream are added. 
[0234] An absolute frame priority (absolute; absolute 
value) is obtained from the sum of a relative frame prior- 
ity and a stream priority (That is. absolute frame priority 
= relative frame priority + stream priority). To obtain an 
absolute frame prtority. it is also possible to use a sub- 
tracting metiiod or a constant-multiplying metiiod. 
[0235] An absolute frame priority mainly uses a net- 
work. This is because the expression using an absolute 
value does not require the necessity for deciding a pri- 
ority for each frame through a repeater such as a router 
or gateway by considering Stream Priority and Frame 
Priority By using the absolute frame priority, such 
processing as disuse of a frame by a repeater is simpli- 
fied. 

[0236] Moreover, it can be expected to apply a relative 
frame priority mainly to an accumulation system for per- 
fomiing recording or editing. In tiie case of an editing 
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operation, a plurality of picture and audio streams may 
be handled at the same time. In this case, the number of 
picture streams or the number of frames that can be 
reproduced may be limited depending on the load of a 
terminal or network. 

[0237] In the above case, it is unnecessary to recalcu- 
late every Frame Priority differently from the case in 
which an absolute value is expressed only by separat- 
ing Stream Priority from Frame Priority, that is, only by 
changing Stream Priority of a stream which an editor 
wants to preferentially display or a user wants to see. 
Thus, it is necessary to use an absolute expression or a 
relative expression in accordance with the purpose. 
[0238] By describing whetiier to use a stream priority 
as a relative value or absolute value, it is possible to 
effectively express a priority for transmission and accu- 
mulation. 

[0239] In the case of the example in Figure 28(b), it is 
differentiated by following a stream priority that the 
value expressed by tiie stream priority is a relative value 
or absolute value by using a flag or identifier for 
expressing whether tiie value expressed by tiie stream 
priority is an absolute value or relative value. In the case 
of a frame priority, a flag or identifier is unnecessary 
because a relative value is described in a communica- 
tion header and an absolute value is described in an 
encoded frame. 

[0240] In tiie case of tiie example in Figure 28(c), a 
flag or identifier for identifying whether a frame priority is 
an absolute value or relative value is used. In the case 
of an absolute value, the frame priority is a priority cal- 
culated In accordance witii a stream priority and a rela- 
tive frame priority and tiierefore, tiie calculation is not 
performed by a repeater or terminal. Moreover, when 
the calculation formula is already known at a terminal, it 
is possible to inversely calculate a relative frame priority 
from an absolute frame priority and a stream priority 
For example, it is also possible to obtain the absolute 
priority (Access Unit Priority) of a packet to be transmit- 
ted from the relational expression 
[0241] "Access Unit Priority = stream priority - frame 
priority". 

In this case, it is also possible to express the frame pri- 
ority as a degradation priority because it is obtained 
after being subtracted from the stream priority 
[0242] Moreover, it is also possible to control data 
processing by relating one stream priority or more to the 
priority for processing of the data passing through the 
logical channel of TCP/IP (port No. of LAN). 
[0243] Furthermore, it is expected tiiat the necessity 
for retransmission can be reduced by assigning a 
stream priority or frame priority lower than that of a 
character or control information to a picture or audio. 
This is because no problem occurs in most cases even 
if a part of a picture or audio is lost. 
[0244] Figure 29 is an illustration for explaining a 
method for assigning a priority to multi-resolution video 
data. 



[0245] When one stream is constituted witii a plurality 
of substreams . it is possitrfe to define a substream 
processing metiiod by adding a stream priority to the 
substreams and describing a logical sum or logical 

5 product under accumulation or transmission. 

[0246] In tiie case of a wavelet, it is possible to decom- 
pose one picture frame into a plurality of different-reso- 
lution picture frames. Moreover, even in tiie case of a 
DCT-base encoding mettiod, it Is possible to deoom- 

10 pose one picture frame into a plurality of different-reso- 
lution picture frames by dividing tiie picture frame into a 
high-frequency component and a low-frequency com- 
ponent and encoding them. 

[0247] In addition to stream priorities added to a plu- 
15 rality of picture streams constituted witii a series of 
decomposed picture frames, the relation between pic- 
ture streams is defined with AND (logical product) and 
OR (logical sum) in order to describe tiie relation. Spe- 
cifically, when the stream priority of a stream A is 5 and 
20 ttiat of a stream B is 10 (tiie smaller a numerical value 
gets, the higher a priority becomes), the relation 
between picture streams is defined that the stream B Is 
disused In tiie case of disuse of stream data depending 
on tiie priority but the sti-eam B is transmitted and proc- 
25 eased witiiout being disused even if the priority of the 
stream B is lower tiian tiie priority of a threshold in tiie 
case of AND by describing tiie relation between 
streams. 

[0248] Thereby, relevant streams can be processed 
30 without being disused. In tiie case of OR, it is defined 
tiiat relevant streams can be disused. It Is possible to 
perform disuse processing at a tiransmitting or receiving 
terminal or a repeating terminal as ever. 
[0249] Moreover, when the same video clip is 
35 encoded to 24 Kbps and 48 Kbps respectively as an 
operator for relational description, there is a case in 
which either 24 or 48 Kbps may be reproduced (exclu- 
sive logical sum EX-OR as relational description). 
[0250] When the priority of tiie former is set to 1 0 and 
40 that of the latter is set to 5, a user can reproduce tiie lat- 
ter in accordance with a priority or select the latter with- 
out following the priority 

[0251 ] Figure 30 is an illustration lor explaining a com- 
munication payload constituting metiiod. 

45 [0252] When constituted with a plurality of sub- 
streams, disuse at a transmission packet level becomes 
easy by. for example, constituting ti^nsmlssion packets 
starting with, for example, one having the highest prior- 
ity in accordance with a stream priority added to a sub- 

50 stream. Moreover, disuse at a communication packet 
level becomes easy by fining grading and uniting the 
information for objects respectively having a high frame 
priority and tiiereby constituting a communication 
packet. 

55 [0253] By relating the sliced structure of a picture to a 
communication packet, return of a missing packet 
becomes easy. That is, by relating the sliced structure of 
a video to a packet structure, a re-sync marker for 
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resynchronization is unnecessary. Unless a sliced 
structure coincides witli the structure of a communica- 
tion packet, it is necessary to add a re-sync marker 
(marker for making a returning position known) so that 
resynchronizatton can be performed if information is 5 
damaged due to a missing packet). 
[0254] In accordance with the above-mentioned, it is 
considered to apply a high error protection to a commu- 
nication packet having a high priority. Moreover, the 
sliced structure of a picture represents the unit of col- 
lected picture information such as GOB or MB. 
[0255] Figure 31 is an illustration for eqslaining a 
method for relating data to communication payload. By 
transmitting a method for relating a stream or object to a 
communication packet together with control Information 
or data, it is possible to generate an optional data format 
in accordance with the communication state or purpose. 
For example, in tiie case of RTP (Real time Transfer 
Protocol), the payload of RTP is defined for each encod- 
ing to be handled. The format of the existing RTP is 
fixed. In the case of H.263, as shown in Figure 31 , tiiree 
data formats from Mode A to Mode C are defined. In the 
case of H.263, a communication payload purposing a 
muHi-resdution picture format is not defined. 
[0256] In the case of the example In Figure 31, Layer 
No. and tiie above relational description (AND, OR) are 
added to the data format of Mode A and defined. 
[0257] Rgure 32 is an illustration for explaining the 
relation between frame priority, stream priority, and 
communication packet priority 
[0258] Moreover, Figure 32 shows an example of 
using a priority added to a communicatfon packet on a 
transmission line as a communication packet priority 
and relating a stream priority and a frame priority to the 
communication packet priority. 
[0259] Generally, in tiie case of communication using 
IP, it is necessary to transmit data by relating a frame 
priority or stream priority added to picture or audio data 
to the priority of a low-order IP packet. Because the pic- 
ture or audio data Is divided into IP packets and trans- 
mitted, it is necessary to relate priorities to each other. 
In the case of the example in Figure 32, because the 
stream priority takes values from 0 to 3 and the frame 
priority takes values from 0 to 5, high-order data can 
take priorities from 0 to 15. 

[0260] In the case of IPv6, priorities (4 bits) from 0 to 
7 are reserved for congestion-controlled traffic. Priori- 
ties from 8 to 15 are reserved for real-time communica- 
tion traffic or not-congestion-controlled traffic. Priority 
15 is the highest priority and priority 8 is tiie lowest pri- 
ority This represents the priority at tine packet level of IR 
[0261] In tiie case of data transmission using IP, it is 
necessary to relate high-order priorities from 0 to 15 to 
low-order IP priorities from 8 to 1 5. To relate priorities to 
each other, it is possible to use a method of clipping 
some of high-order priorities or relate priorities to each 
otiier by using a performance function. Relating of high- 
order data with a low-order IP priority is performed at a 



repeating node (router or gateway) or transmitting and 
receiving terminals. 

[0262] Transmitting means is not restricted to only IP. 
It is possible to use a transmission pactet having a flag 
showing whetiier it can be disused like TS (transport 
stream) of ATM or MPEG2. 

[0263] The frame priority and stream priority having 
been described so far can be applied to a transmitting 
medium or data-recording medium. It is possible to use 
a floppy disk or optical disk as a data-recording 
medium. 

[0264] Moreover, it is possible to use not only tiie 
floppy disk or optical disk but also a medium such as an 
IC card or ROM cassette as long as a program can be 
recorded in the medium. Furtiiermore, it is possible to 
use an audio-video repeater such as a router or gate- 
way for relaying data. 

[0265] Furtiiermore, preferential retransmission is 
realized by deciding time-series data to be retransmit- 
ted in accordance witii tiie Information of Stream Prior- 
ity (inter-time-series-data priority) or Frame Priority 
(intra-time-series-data priority). For example, when 
decoding is performed at a receiving terminal in accord- 
ance with priority information, it is possible to prevent a 
stream or frame tiiat is not an object for processing from 
being retransmitted. 

[0266] Furthermore, separately from a present priority 
to be processed, it is possible to decide a stream or 
frame having a priority to be retransmitted in accord- 
ance witii tiie relation between retransmission fre- 
quency and suocessful transmission frequency. 
[0267] Furthermore, in the case of a transmitting-side 
terminal, preferential transmission is realized by decid- 
ing time-series data to be transmitted in accordance 
witii the information of Sfream Priority (ii^^r-time- 
series-data priority) or Frame Priority (intra-time-series- 
data priority). For example, by deciding the priority of a 
stream or frame to be transmitted in accordance with an 
average transfer rate or retransmission frequency, it is 
possible to transmit an adaptive picture or audio even 
when a network Is overloaded. 
[0268] The above embodiment is not restricted to two- 
dimensional-picture synthesis. It is also possible to use 
an expression method obtained by combining a two- 
dimensional picture with a three-dimensional picture or 
Include a picture-syntiiesizing metiiod for synthesizing 
a plurality of pictores so as to be adjacent to each other 
like a wide-visual-field picture (panorama picture). 
Moreover, communication systems purposed by the 
present invention are not restricted to bidirectional 
CATV or B-ISDN. For example, transmission of pictjres 
and audio from a center-side terminal to a house-side 
terminal can use radio waves (e.g. VHF band or UHF 
band) or satellite broadcasting and information origina- 
tion from the house-side terminal to the center-side ter- 
minal can use an analog telephone line or N-ISDN (it is 
not always necessary that pictures, audio, or data are 
multiplexed). Moreover, it is possible to use a communi- 
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cation system using radio such as an IrDA, PHS (Per- 
sonal Handy Phone) or radio LAN. 
[0269] Furthermore, a purpose terminal can be a port- 
able terminal such as a portable information temiinal or 
a desktop terminal such as a set-top BOX or personal s 
computer. 

[0270] As described above, the present Invention 
makes it easy to handle a plurality of video streams and 
a plurality of audio streams and mainly synchronise and 
reproduce important scene cut together with audio by 
reflecting tiie intention of an editor. 
[0271] An embodiment of tiie present invention is 
described below by referring to tiie accompanying draw- 
ings. 

[0272] The embodiment desaibed below solves any 
one of the above problems (CI) to (C3). 
[0273] Rgure 33 shows tiie structure of the transmitter 
of tiie first ent)odiment. Symbol 2101 denotes a picture- 
input terminal and the size of a sheet of picture has 1 44 
pixels by 176 pixels. Symbol 2102 denotes a vkJeo 
encoder that is constituted with four components 1 021 . 
1022. 1023, and 1024 (see Recommendation H.261). 
[0274] Symbol 1 021 denotes a switching unit for divid- 
ing an input picture into macroblocks (a square region of 
16 pixels by 16 pixels) and deciding whetiier to intra- 
encode or inter-encode the blocks and 1022 denotes 
movement compensating means for generating a move- 
ment compensating picture in accordance witii tiie local 
decoded picture which can be calculated in accordance 
witii tiie last-time encoding result, calculating tiie differ- 
ence between the nfKsvement compensating picture and 
an input pk:ture, and outputting the result in macrob- 
locks. Movement compensation includes halfpixel pre- 
diction having a long processing time and fullpixel 
prediction having a short processing time. Symbol 1023 
denotes ortiiogonal transforming means for applying 
DOT transformation to each macroblock and 1024 
denotes variable-length-encoding means for applying 
entropy encoding to the DCT transformation result and 
other encoded information. 

[0275] Symbol 2103 denotes counting means for 
counting execution frequencies of four components of 
the video encoder 2102 and outputting tiie counting 
result to transforming means every input picture. In this 
case, the execution frequency of the halfpixel prediction 
and tiiat of tiie fullpixel prediction are counted from the 
movement compensating means 1022. 
[0276] Symbol 21 04 denotes transforming means for 
outputting the data string shown in Figure 34. Symbol 
2105 denotes transmitting means for multiplexing a var- 
iable-length code sent from the video encoder 21 02 and 
a data string sent from the transforming means 2104 
into a data string and outputting the data string to a data 
output terminal 2109. 

[0277] According to the above structure, it is possible 
to transmit the execution frequencies of indispensable 
processing (switching unit 1021, orthogonal transform- 
ing means 1023, and variable-length encoding means 



1 024) and dispensable processing (nrxyvement compen- 
sating means 1022) to a receiver. 
[0278] The transmitter of tiie first embodiment conre- 
sponds to claim 68. 

[0279] Figure 40 is a flowchart of tiie transmitting 
method of the second embodiment 
[0280] Because operations of this embodiment are 
similar to those of the first embodiment, corresponding 
elements are added. A picture is input in step 801 (pic- 
ture input terminal 2101) and the pkrture is divided into 
macrotrfocks in step 802. Hereafter, processings from 
step 803 to step 806 are repeated until the processing 
corresponding to every macroblock is completed in 
accordance witii the conditional branch in step 807. 
Moreover, when each processing is executed so ttiat 
frequencies of tiie processings from step 803 to step 
806 can be recorded in specific variables, a conrespond- 
ing varlak>le is incremented by 1. 
[0281] First, it is deckled whether to intra-encode or 
inter-encode a macroblock to be processed in step 803 
(switching unit 1021). When inter-encoding the macrob- 
lock. movement compensation is performed in step 804 
(movement compensating means 1022). Thereafter, 
DCT transformation and variable-length encoding are 
performed in steps 805 and 806 (ortiiogonal transform- 
ing means 1023 and variable-length encoding means 
1024 ). When processing for every macroblock is com- 
pleted (in tiie case of Yes in step 807). tiie variable 
showing tiie execution frequency conresponding to each 
processing is read in step 808. tiie data string shown In 
Rgure 2 Is generated, and the data string and a code 
are multiplexed and output. The processings from step 
801 to step 808 are repeatedly executed as long as 
input pictures are continued. 

[0282] The above structure makes it possible to trans- 
mit the execution frequency of each processing. 
[0283] The transmitting method of the second embod- 
iment corresponds to claim 67. 
[0284] Figure 35 shows the structure of the receiver of 
the third embodiment. 

[0285] In Figure 35. symbol 307 denotes an input ter- 
minal for inputting tiie output of tiie transmitter of the 
first embodiment and 301 denotes receiving means for 
fetching a variable-length code and a data string 
through Inverse multiplexing in accordance with the out- 
put of the transmitter of the first embodiment and out- 
putting tiiem. In this case, it is assumed that the time 
required to receive tiie data for one sheet is measured 
and also output. 

[0286] Symbol 303 denotes a decoder for a video 
using a variable-length code as an input, which is con- 
stituted with five components. Symbol 3031 denotes 
variable-length decoding means for fetching a DCT 
coefficient and other encoded information from a varia- 
ble-length code, 3032 denotes inverse orthogonal trans- 
forming means for applying Inverse DCT transformation 
to a DCT coefficient, and 3033 denotes a switching unit 
for switching an output to upskJe or downskle every 
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macroblock in accordance with the encoded information 
showing whether the macroblock is intra-encoded or 
inter-encoded. Symbol 3034 denotes movement com- 
pensating means for generating a movement compen- 
sating picture by using the last-time decoded picture 
and movement encoded information, and adding and 
oufputting the outputs of the inverse orthogonal trans- 
forming means 3032. Symbol 3035 denotes execution- 
time measuring means fbr measuring and outputting the 
execution time until decoding and outputting of a picture 
is completed after a variable-length code is input to the 
decoder 303. 

[0287] Symbol 302 denotes estimating means fbr 
receiving the execution frequency of each element (var- 
iable-length decoding means 3031. inverse orthogonal 
transforming means 3032. switching unit 3033, or 
movement compensating means 3034) from a data 
string sent from the receiving means 301 and execution 
time from the execution-time measuring means 3035 to 
estimate the ^cecution time of each element. 
[0288] To estimate the execution time of each ele- 
ment, it is posslt^le to use the linear regression and 
assume an estimated execution time as a purposed var- 
iable y and the execution frequency of each component 
as an explanatory variable xoi. In this case, it may be 
possible to regard a regression parameter ax>\ as the 
execution time of each element. Moreover, in the case 
of linear regression, it is necessary to accumulate 
much-enough past data and resuHantly, many memo- 
ries are wasted. However, to avoid many memories from 
being wasted, it is also possible to use the estimation of 
an internal-state variable by a Kalman filter. It is possi- 
ble to consider the above case as a case in which an 
observed value is assumed as an execution time, the 
execution time of each element is assumed as an inter- 
nal-state variable, and an observation matrix C changes 
every step due to the execution frequency of each ele- 
ment. Symbol 304 denotes frequency reducing means 
fbr changing the execution frequency of each element 
so as to reduce the execution frequency of fullpixel pre- 
diction and increase the execution frequency of halfpi^l 
prediction by a con^esponding value. The method fbr 
calculating the corresponding value is shown below. 
[0289] First, the execution frequency and estimated 
execution time of each element are received from the 
estimating means 302 to estimate an execution time. 
When the execution time exceeds tiie time required to 
receive the data from tiie receiving means 301 , the exe- 
cution frequency of fullpixel prediction is increased and 
the execution frequency of halfplxel prediction is 
decreased until tiie former time does not exceed the lat- 
ter time. Symbol 306 denotes an output terminal fbr a 
decoded picture. 

[0290] Moreover, there is a case in which the move- 
ment compensating means 3034 Is designated so as to 
perform hatfpixel prediction in accordance with encoded 
information. In tiiis case, when tiie predetermined exe- 
cution frequency of halfpixel prediction is exceeded, a 



halfpixel movement is rounded to a fullpixel movement 
to execute fullpixel prediction. 
[0291 ] According to tiie above-described first and third 
embodiments, tiie execution time of decoding is esti- 

5 mated in accordance with tiie estimated execution time 
of each element and, when the decoding execution time 
may exceed the time (designated time) required to 
receive the data fbr one sheet halfpixel prediction hav- 
ing a long execution time is replaced with fullpi^l pre- 

10 diction. Thereby, it is possible to prevent an execution 
time from exceeding a designated time and solve tiie 
problem (CI) (corresponding to clainrs 68 and 74). 
[0292] Moreover, a case of regarding the parts of 
indispensable and dispensable processings as two 

IS groups oon^esponds to claims 66 and 72 and a case of 
regarding the part of a video as waveform data con-e- 
sponds to claims 64 and 70. 

[0293] Furthermore, by using no high-frequency com- 
ponents in tiie IDCT calculation by a receiver, it is pos- 

20 sible to reduce the processing time for the IDCT 
calculation. That is. by regarding tiie calculation of low- 
frequency components as indispensable processing 
and tiie calculation of high-frequency components as 
dispensable processing in the IDCT calculation, it is 

25 also possible to reduce the calculation frequency of 
high-frequency components in the IDCT calculation. 
[0294] Figure 41 is a flowchart of the receiving method 
of the fourtii embodiment. 

[0295] Because operations of tiiis embodiment are 

30 similar to those of the tiiird embodiment, con-esponding 
elements are added. In step 901, the variable aj fbr 
repressing the execution time of each element is initial- 
ized (estimating means 302) . In step 902, multiplexed 
data Is input and tiie time required fbr multiplexing tiie 

35 data Is measured (receiving means 301) . In step 903, 
the multiplexed data is divided into a variable-lengfli 
code and a data string and output (receiving means 
301). In step 904, each execution frequency is fetched 
from a data string (Figure 2) and it Is set to xj. In step 

40 905, an actual e)«cution frequency is calculated in 
accordance with tiie execution time aJ of each element 
and each execution frequency xJ (frequency reducing 
means 304). In step 906, measurement of tiie execution 
time for decoding Is started. In step 907, a decoding 

45 routine to be described later Is started. Thereafter, in 
step 908, measurement of the decoding execution time 
is ended (video decoder 303 and execution-time meas- 
uring means 3035). In step 908, the execution time of 
each element is estimated in accordance with the 

50 decoding execution time in step 908 and the actual exe- 
cution frequency of each element in step 905 to update 
aJ (estimating means 302). The above processing is 
executed every input multiplexed data. 
[0296] Moreover, in step 907 for decoding routine, var- 

55 iable-length decoding Is performed In step 910 (varia- 
ble-length decoding means 3031), inverse ortiiogonal 
transfomrtation is performed in step 91 1 (inverse orthog- 
onal transforming means 3032), and processing is 
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branched in step 91 2 in accordance with the information 
of the lntra-/inter-processing fetched through the 
processing in step 91 0 (switching unit 3033). In the case 
of inter-processing, movement compensation is per- 
formed in step 913 (movement compensating means 
3034). In step 913, the execution frequency of halfpixel 
prediction is counted in step 913. When the counted 
execution frequency exceeds the actual execution fre- 
quency obtained in step 905, halfpixel prediction is 
replaced with fullpixel prediction for execution. After the 
above processing is applied to every macroblock (step 
91 4). the routine is ended. 

[0297] According to the above-described second and 
fourth embodiments, the execution time of decoding is 
estimated in accordance with the estimated execution 
time of each element and, when the execution time may 
exceed the time required to receive the data for one 
sheet (designated time), haHjpixel prediction having a 
long execution time is replaced with fullpixel prediction. 
Thereby, it is possible to prevent an execution time from 
exceeding a designated time and solve the problem 
(CI) (con-esponding to claims 67 and 73). 
[0298] Furthermore, a case of regarding the parts of 
dispensable and Indispensable processings as two 
groups corresponds to claims 65 and 71 and a case of 
regarding the part of a video as waveform data corre- 
sponds to claims 63 and 69. 
[0299] Rgure 36 shows the structure of the receiver of 
the fifth embodiment 

[0300] Most components of this embodiment are the 
same as those described for the second embodiment. 
However, two added components and one connected 
component are described below. 
[0301] Symbol 402 denotes estimating means 
obtained by correcting the estimating means 302 
described for the second embodiment so as to output 
the execution time of each element obtained as the 
result of estimation separately from an output to fre- 
quency limiting means 304. Symbol 408 denotes trans- 
mitting means for generating the data string shown in 
Figure 37 in accordance with the execution time of each 
element and outputting it. When expressing an execu- 
tion time with 16 bits by using microsecond as the unit, 
up to approx. 65 msec can be expressed. Therefore, 
approx. 65 msec will be enough. Symbol 409 denotes 
an output terminal for transmitting the data string to 
transmitting means. 

[0302] Moreover, a receiving method corresponding to 
the fifth embodiment can be obtained only by adding a 
step for generating the data string shown in Rgure 37 
immediately after symbol 808 In Figure 40. 
[0303] Rgure 38 shows the structure of the transmitter 
of the sixth embodiment. 

[0304] Most components of this embodiment are the 
same as those described for the first embodiment. How- 
ever, two added components are described below. Sym- 
bol 606 denotes an input terminal for receiving a data 
string output by the receiver of the third embodiment 



and 607 denotes receiving means for receiving the data 
string and outputting the execution time of each ele- 
ment. Symbol 608 denotes deciding means for obtain- 
ing the execution frequency of each element and its 

5 obtaining procedure is described below. Rrst, every 
macroblock in a picture is processed by the switching 
unit 1021 to obtain the execution frequency of the 
switching unit 1021 at this point of time. Moreover, it is 
possible to uniquely decide execution frequencies by 

10 the movement compensating means 1022, orthogonal 
transforming means 1023, and variable-length encod- 
ing means 1024 in accordance with the processing 
result up to this point of time. Therefore, the execution 
time required for decoding at the receiver side is esti- 

15 mated by using these execution frequencies and the 
execution time sent from the receiving means 607. The 
estimated decoding time is obtained as the total sum of 
the product between the execution time and execution 
frequency of each element every element Moreover, 

20 when the estimated decoding time is equal to or more 
than the time required to transmit the number of codes 
(e.g. 16 Kbits) to be generated through this picture des- 
ignated by a rate controller or the like (e.g. 250 msec 
when a transmission rate Is 64 Kbits/sec), the execution 

25 frequency of fullpixel prediction is increased and the 
execution frequency of hatfpixei predlctton is decreased 
so that the estimated decoding execution time does not 
exceed the time required for transmission. (Because 
fullpixel prediction has a shorter execution time, it is 

30 possible to reduce the execution time of fullpixel predic- 
tion by reducing the frequency of fullpixel prediction.) 
[0305] Moreover, tiie video encoder 2102 performs 
various processings in accordance with tiie execution 
frequency designated by tiie deciding means 608. For 

35 example, after the movement compensating means 
1022 executes halfpixel prediction by the predetermined 
execution frequency of halfpixel prediction, it executes 
only fullpixel prediction. 

[0306] Furthermore, it is possible to improve the 

40 selecting metiiod so that halfpixel prediction Is uniformly 
dispersed In a picture. For example, it is possible to use 
a method of first obtaining every macroblock requiring 
halfpixel prediction, calculating the product (3) obtained 
by dividing the number of the above macroblocks (e.g. 

45 12) by the execution frequency of halfpixel prediction 
(e.g 4), and applying halfpixel prediction only to a mac- 
roblock whose sequence from the beginning of the mac- 
roblocks requiring halfpixel prediction is divided by the 
above product witiiout a remainder (0, 3. 6. or 9). 

50 [0307] According to the above-described fifth and 
sixtti embodiments , the execution time of each esti- 
mated element is transmitted to the transmitting side, 
the execution time of decoding is estimated at the trans- 
mitting side, and halfpixel prediction having a long exe- 

55 cution time is replaced with fullpixel prediction so that 
the estimated decoding execution time does not exceed 
tiie time (designated time) probably required to receive 
the data for one sheet. Thereby, the information for half- 
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pixel prediction among the sent encoded information is 
not disused and thereby, it Is possible to prevent an exe- 
cution time from exceeding a designated time and solve 
the problem (C2) (corresponding to claims 76 and 78). 
[0306] Moreover, in the case of dispensable process- s 
ing, it is possible to divide inter-macroblock encoding 
into such three movement compensations as normal 
nrK>vement compensation, 8x8 movement compensa- 
tion, and overlap movement compensation. 
[0309] Rgure 42 is a flowchart of the transmitting io 
method of the seventh embodiment. 
[0310] Because operations of this embodiment are 
similar to those of the sixth embodiment, corresponding 
elements are added. In step 1 001 , the initial value of the 
execution time of each processing is set. A picture is is 
input (input terminal 2101) in step 801 and it is divided 
into macroblocks in step 802. In step 1002, it is decided 
whether to intra-encode or inter-encode every macrob- 
lock (switching unit 1021). Resultantly, the execution 
frequency of each processing from step 1005 to step so 
806 is known. Therefore, in step 1003. an actual execu- 
tion frequency is calculated in accordance with the 
above execution frequency and the execution time of 
each processing (deciding means 608). 
[0311] Hereafter, the processings from step 1005 to 2s 
step 806 are repeated until the processing for every 
macroblock is completed in accordance with the condi- 
tional branch in step 807. 

[0312] Moreover, when each processing is executed, 
a corresponding variable is incremented by 1 so that the so 
processing frequencies from step 1005 to step 806 can 
be recorded in a specific variable. Rrst. in step 1005, 
branching is performed in accordance with the decision 
result in step 1002 (switching unit 1021). In the case of 
irrter-encoding, movement compensation is performed ss 
in step 804 (movement compensating means 1022). In 
this case, the frequency of halfpixel prediction is 
counted. When the counted frequency exceeds the 
actual frequency obtained in step 1003, fullpixel predic- 
tion is executed instead without executing halfpixel pre- 40 
diction. Thereafter, in steps 805 and 806, DOT 
transformation and variable-length encoding are per- 
formed (orthogonal transforming means 1023 and vari- 
able-length encoding means 1024). When the 
processing for every macroblock is completed, (in the 45 
case of Yes in step 807), the variable showing tiie exe- 
cution frequency corresponding to each processing is 
read in step 808, tiie data string shown In Figure 2 is 
generated, and the data string and a code are multi- 
plexed and output. In step 1004. tiie data string is so 
received and the execution time of each processing is 
fetched from the data string and set. 
[0313] Processings from step 801 to step 1004 are 
repeatedly executed as long as pictures are input. 
[031 4] According to the paragraph beginning with the ss 
final "Moreover" of the descriptive portion of the fifth 
embodiment and tiie seventii embodiment tiie esti- 
mated ^cecution time of each element is transmitted to 



the transmitting side, the execution time of decoding is 
estimated at the transmitting side, and halfpixel predic- 
tion having a long execution time is replaced with full- 
pixel prediction so tiiat tiie estimated decoding 
execution time does not exceed ttie time (designated 
time) probably required to receive tiie data for one 
sheet. Thereby, the information for halfpixel prediction 
among tiie sent encoded information is not disused and 
rt is possible to prevent tiie execution time from exceed- 
ing the designated time and solve the problem (C2) 
(con^esponding to claims 75 and 77). 
[031 5] Figure 39 shows tiie structure of tiie transmit- 
ting apparatus of tiie eightti embodiment of the present 
invention. 

[031 6] Most components of this embodiment are tiie 
same as tiiose desaibed for the first embodiment. 
Therefore, four added components are described 
below. 

[0317] Symbol 7010 denotes execution-time measur- 
ing means for measuring the executk}n time until encod- 
ing and outputting of a picture are completed after tiie 
picture is input to an encoder 2102 and outputting tiie 
measured execution time. Symbol 706 denotes estimat- 
ing means for receiving execution frequencies of ele- 
ments (switching unit 1021, nrKivement compensating 
means 1022, ortiiogonal transforming means 1023, and 
variable-length decoding means 1024) of a data string 
from counting means 2103 and the execution time from 
the execution-time measuring means 7010 and estimat- 
ing tiie execution time of each element. It is possible to 
use an estimating method same as that described for 
the estimating means 302 of tiie second embodiment. 
Symbol 707 denotes an input terminal for inputting a 
frame rate value sent from a user and 708 denotes 
deciding means for obtaining the ^cecution frequency of 
each element The obtaining procedure is described 
below. 

[0318] First, every maaoblock in a picture is proc- 
essed by the switching unit 1021 to obtain the execution 
frequency of the switching unit 1 021 at this point of time. 
Thereafter, it is possible to uniquely decide execution 
frequencies by the movement compensating means 
1022, orthogonal transforming means 1023, and varia- 
ble-length encoding means 1024 in accordance wrth the 
processing result up to this point of time. Then, the total 
sum of products between tiie execution frequency and 
the estimated execution time of each element sent from 
tiie estimating means 706 is obtained every element to 
calculate an estimated encoding time. Then, when the 
estimated encoding time is equal to or longer tiian the 
time usable for encoding of a sheet of picture obtained 
from tiie inverse number of the frame rate sent from 
symbol 707. the execution frequency of fullpixel predic- 
tion is increased and that of halfpixel prediction Is 
decreased. 

[0319] By repeating the above change of execution 
frequencies and calculation of the estimated encoding 
time until the estimated encoding time becomes equal 
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to or shorter than the usable time, each execution fre- 
quency is decided. 

[0320] Moreover, the video encoder 2102 performs 
various processings In accordance with the execution 
frequency designated by the deciding means 608. For s 
example, after the nwvement compensating means 
1022 executes halfpixel prediction by the predetermined 
execution frequency of halfi3tol prediction, It executes 
only fullpixel prediction. 

[0321] Furthermore, ft Is also possible to Improve a 
selecting method so that haKjpixel prediction is uniformly 
dispersed in a picture. For example, it is possible to use 
a method of obtaining every macroblock requiring half- 
pixel prediction, calculating the product (3) obtained by 
dividing the number of macrobtod® requiring haHpixel 
prediction (e.g. 12) by the execution frequency of half- 
pixel prediction (e.g. 4), and applying halfpixel predic- 
tion only to a macroblock whose sequence from the 
beginning of the macroblocks requiring halfpixel predic- 
tion is divided by the product without remainder (0. 3. 6, 
or 9). 

[0322] The above eighth embodiment makes it possi- 
ble to solve the problem (C3) by estimating the execu- 
tion time of each processing, estimating an execution 
time required for encoding in accordance with the esti- 
mated execution time, and deciding an execution fre- 
quency so tiiat tiie estimated encoding time becomes 
equal to or shorter than the time usable for encoding of 
a picture determined in accordance with a frame rate 
(corresponding to daim 80). 
[0323] Moreover, because the movement compensat- 
ing means 1022 detects a movement vector, tiiere Is a 
full-search movement-vector detecting method for 
detecting a vector for minimizing SAD (sum of absolute 
values of differences every pixel) among vectors In a 
range of 15 horizontal and vertical pixels. Furtiiermore, 
there is a three-step movement-vector detecting 
method (described in annex of H.261). The three-step 
movement-vector detecting method executes the 
processing of selecting nine points uniformly an-anged 
in the above retrieval range to select a point having a 
minimum SAD and tiien, selecting nine points again in a 
narrow range close to the above point to select a point 
having a minimum SAD one more time. 
[0324] It is also possible to properly decrease tiie exe- 
cution frequency of tiie full-search movement-vector 
detecting method and properly increase tiie execution 
frequency of tiie three-step movement-vector detecting 
method by regarding these two methods as a dispensa- 
ble processing method and estimating the execution 
time of each of tiie two methods, estimating an execu- 
tion time required for encoding in accordance with tiie 
estimated execution time so tiiat the estimated execu- 
tion time becomes equal to or shorter than the time des- 
ignated by a user. 

[0325] Moreover, it is possible to use a movement- 
vector detecting m^hod using a fixed retrieval fre- 
quency and furtiier simplifying tiie processing or a 



movement-vector detecting metiiod of returning only 
tiie movement vector (0, 0) as a result together with the 
tiiree-step movement-vector detecting method. 
[0326] Rgure 43 is a flowchart of tiie transmftting 
metiiod of tiie nintii embodiment. 
[0327] Because operations of tiiis emtxxJiment are 
similar to tiiose of tiie eightii embodiment, correspond- 
ing elements are added. For the detailed operation in 
each flow, refer to the desalption of corresponding ele- 
ments. 

[0328] Moreover, because this embodiment is almost 
the same as the second embodiment, only different 
points are explained below. 

[0329] In step 1 101 , tiie initial value of tiie execution 
time of each processing Is set to a variable aj. Moreo- 
ver, in step 1102, a frame rate is input (input terminal 
707). In step 1103, an actual execution frequency is 
decided in accordance wHh tiie frame rate and the exe- 
cution time a J of each processing in step 1 102 and the 
execution frequency of each processing obtained from 
the inti'a-/inter-processing decision result in step 1002 
(deckJing means 708). In steps 1 105 and 1 106, the exe- 
cution time of encoding is measured. In step 1 104. the 
execution time of each processing is estimated in 
accordance witii the execution time obtained In step 
1106 and tiie actual execution frequency of each 
processing to update tiie variable aJ (estimating means 
706). 

[0330] According to tiie above-described nintii 
embodiment, the execution time of each processing is 
estimated and an execution time required for encoding 
Is previously measured in accordance with the esti- 
mated execution time. Thus, it is possible to solve the 
problem (03) by deciding an actual execution frequency 
so tiiat the estimated encoding time becomes the time 
usable for tiie encoding of a picture determined in 
accordance with a frame rate or shorter (corresponding 
to claim 79). 

[0331 ] In the case of the second embodiment, it is also 
possible to add a two-byte region immediately after the 
start code shown in Figure 2 when the data string Is 
generated in step 808 and add the binary notation of a 
code length to the region. 

[0332] Moreover, in the case of the fourth embodi- 
ment, it is also possible to extract a code length from the 
two-byte region when multiplexed data is input in step 
902 and use tiie code transmission time obtained from 
the code length and tiie code transmission rate for the 
execution frequency calculation in step 905 (the execu- 
tion frequency of halfpixel prediction is decreased so as 
not to exceed the code transmission time). This corre- 
sponds to claims 81 and 83. 

[0333] Furtiiermore. in the case of the first embodi- 
ment, it is also possible to add a two-byte region imme- 
diately after the start code shown in Figure 2 when a 
data string is generated in step 2104 and add tiie binary 
notation of a code length to tiie region. 
[0334] Furthermore, in the case of tiie third embodl- 
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ment, it is also possible to extract a code length from the 
two-byte region when multiplexed data is input in step 
301 and use a code transmission time obtained from the 
code length and the code transmission rate for the exe- 
cution frequency calculation in step 304 (the execution 
frequency of halfpixel prediction is decreased so as not 
to exceed the code transmission time). This corre- 
sponds to claims 82 and 84. 
[0335] Furthermore, in the case of the fourth embodi- 
ment, an actual execution frequency of halfpixel predic- 
tion is recorded immediately after step 909 to calculate 
a maximum value. When the maximum value is equal to 
or less than a small-enough value (e.g. 2 or 3). rt is also 
possible to generate a data string (data string compris- 
ing a specific bit pattern) showing that halfpixel predic- 
tion is not used and transmit the generated data string. 
Furthermore, in the case of the second emtxxdiment, rt 
is confirmed whether the data string is received immedi- 
ately after step 808 and when the data string showing 
that halfpixel predictfon is not used is received, rt is also 
possible to make movement compensation processing 
always serve as fullpixel prediction in step 808. This cor- 
responds to claims 93 and 91 . 
[0336] Furthermore, the above concept can be 
applied to cases other than movement compensation. 
For example, it is possible to reduce tiie DCT calcula- 
tion time by using no high-frequency component for 
DCT calculation. That is, in the case of a receiving 
method, when tiie rate of the IDCT-calculation execution 
time to the entire execution time exceeds a certain 
value, a data string showing that tiie rate mceeds a cer- 
tain value is transmitted to the transmitting side. When 
the transmitting side receives the data string, it is also 
possible to calculate only low-frequency components 
through tiie DCT calculation and decrease all high-fre- 
quency components to zero. This corresponds to daim 
89. 

[0337] Furthermore, ttiough the embodiment is 
described above by using a picture, it is possible to 
apply each of the above methods to audio instead of 
video. This corresponds to claims 85 and 87. 
[0338] Furtiiermore. in tiie case of the third embodi- 
ment, an actual execution frequency of halfpixel predic- 
tion is recorded in step 3034 to calculate a maximum 
execution frequency. Then, when the maximum value is 
a small-enough value or less (e.g. 2 or 3). it is possible 
to generate and transmit a data string showing tiiat half- 
pixel prediction is not used (data string comprising a 
specific bit pattern). Furthermore, in tiie case of the first 
embodiment, when receiving a data string showing that 
halfpixel prediction is not used. It is possible to make the 
movement compensation processing in step 1022 
always serve as fullpixel prediction. This corresponds to 
claims 94 and 92, 

[0339] Furthermore, the above concept can be 
applied to cases otiier than movement compensation. 
For example, by using no high-frequency component for 
DCT calculatfon, it is possible to reduce tiie DCT calcu- 



lation processing time. That is, in the case of a receiving 
method, when the rate of IDCT-calculation execution 
time to tiie entire execution time exceeds a certain 
value, a data string showing that the rate exceeds a cer- 

5 tain value is transmitted to the transmitting side. 

[0340] When the transmitting side receives tiie data 
string, it is possible to calculate only low-frequency 
components tiirough tiie DCT calculation and reduce all 
high-frequency components to zero. This corresponds 

10 to claim 90. 

[0341] Furthermore, though the embodiment is 
described above by using a picture, it is also possible to 
apply the above metiiod to audio instead of picture. This 
corresponds to claims 86 and 88. 

IS [0342] As described above, according to claims 68 
and 74 (e.g. first and tiiird embodiments), the execution 
time of decoding is estimated in accordance witti the 
estimated execution time of each element and, when 
the estimated decoding 6)®cution time may exceed the 

20 time (designated time) required to receive the data for 
one sheet, halfpixel prediction having a long execution 
time is replaced witii fullpixel prediction. Thereby, it is 
possible to prevent the execution time from exceeding 
tiie designated time and solve the problem (CI). 

25 [0343] Furtiiermore, according to claims 75 and 77 
(e.g. fifth and seventh embodiments), the estimated 
execution time of each element is transmitted to the 
transmitting side, tiie execution time of decoding is esti- 
mated at the transmitting side, and halfpixel prediction 

30 having a long execution time is replaced with fullpixel 
prediction so tiiatthe estimated decoding time does not 
exceed the time (designated time) probably required to 
receive tiie data for one sheet. Thereby, the information 
for halfpixel prediction in the sent encoded information 

3s Is not disused and it is possible to prevent tiie execution 
time from exceeding tiie designated time and solve tiie 
problem (C2). 

[0344] Furthermore, according to claim 79 (e.g. ninth 
embodiment), it is possible to solve the problem (C3) by 

40 estimating the execution time of each processing, nfx>re- 
over estimating the execution time required for encoding 
in accordance with the estimated execution time, and 
deciding an executing frequency so tiiat the estimated 
encoding time becomes equal to or less than the time 

45 usable for encoding of a picture decided in accordance 
witii a frame rate. 

[0345] Thus, the present invention makes it possible 
to realize a function (CGD: Computational Graceful 
Degradation) for slowly degrading quality even if a cal- 
50 culated load increases and thereby, a very large advan- 
tage can be obtained. 

[0346] Moreover, it is possitrfe to perform operations 
same as described above by a computer by using a 
recording medium such as a magnetic recording 
55 medium or optical recording medium in which a pro- 
gram for making the computer execute all or part (or 
operations of each means) of the each steps (or each 
means) desaibed in any one of tiie above-described 
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embodiments. 
Industrial Applicability 

[0347] As described above, the present invention s 
makes it possible to change information frames corre- 
spondingly to the situation, purpose, or transmission 
line by dynamically deciding the frames of data control 
information, transmission control infbrmatioa and con- 
trol infonmation used for transmitting and receiving ter- io 
minals. Moreover, it is easy to handle a plurality of video 
streams or a plurality of audio streams and mainly 
reproducing an important scene cut synchronously with 
audio by reflecting the intention of an editor. Further- 
more, it is possible to prevent an execution time from is 
exceeding a designated time by estimating the execu- 
tion time of decoding in accordance witii tiie execution 
time of each estimated element and replacing halfpixel 
prediction having a long execution time witii fullpixel 
prediction when tiie estimated decoding execution time so 
may exceed the time (designated time) required to 
receive tiie data for one sheet. 

Claims 

25 

1. An audio-video transmitting apparatus comprising 
transmitting means for transmitting the content con- 
cerned with a transmitting metiiod and/or the struc- 
ture of data to be transmitted or an identifier 
showing the content as transmission format infbr- so 
mation tiirough a transmission line same as that of 
the data to be transmitted or a transmission line dif- 
ferent from tiie data transmission line; wherein 

said data to be transmitted is video data and/or as 
audio data. 

2. The audio-video transmitting apparatus according 
to claim 1, wherein said transmission format infor- 
mation is included in at least one of data control 40 
information added to said data to control said data, 
transmission control information added to said data 

to transmit said data, and information for controlling 
the processing of the terminal side. 

45 

3. The audio-video transmitting apparatus according 
to claim 2, wherein at least one of said data control 
information, transmission control information, and 
information for controlling the processing of said 
terminal side in dynamically changed. so 

4. The audio-video transmitting apparatus according 
to claim 3, wherein 

said data is divided into a plurality of packets, ss 
and 

said data control information or said transmis- 
sion control information is added not only to the 



head packet of said divided packets but also to 
a middle packet of them. 

5. The audio-video transmitting apparatus according 
to daim 1 , wherein an identifier showing whether to 
use timing information concerned with said data as 
information showing tiie reproducing time of said 
data is included in said transmission format infor- 
mation. 

6. The audio-video transmitting apparatus according 
to claim 1, wherein said ti'ansmission format infor- 
mation is the structural information of said data and 
a signal which is output from a receiving apparatus 
receiving the transmitted structural information of 
said data and which can be received is confirmed 
and thereafter, said transmitting means transmits 
conresponding data to said receiving apparatus. 

7. The audio-video transmitting apparatus according 
to claim 1, wherein said fransmisslon format infor- 
mation Include (1) an identifier for kientifying a pro- 
gram or data to be used by a receiving apparatus 
later and (2) at least one of a flag, counter, and 
timer as information for knowing tiie point of time in 
which said program or data is used or tiie term of 
valkjity for using said program or data. 

8. The audio-vkieo transmitting apparatus according 
to claim 7, wherein said point of time in which said 
program or data is used is transmitted as transmis- 
sion control information by using a transmission 
serial number for identifying a transmission 
sequence or as information to be transmitted by a 
packet different from tiiat of data to control terminal- 
side processing. 

9. The audio-video fransmltting apparatus according 
to claim 2 or 3. wherein 

storing means for storing a plurality of contents 
concerned with said transmitting method 
and/or said structure of data to be transmitted 
and a plurality of its identifiers are included, 
and 

said identifier is included in at least one of said 
data control information, transmission control 
information, and information for controlling ter- 
minal-side processing as said transmission for- 
mat information. 

10. The audio-vkleo transmitting apparatus according 
to claim 2 or 3, wherein storing means for storing a 
plurality of contents concerned witii said transmit- 
ting method and/or said structure of data to be 
transmitted are included, and 

said contents are included in at least one of 
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said data control information, transmission 
control information, and information for control- 
ling terminal-side processing as said transmis- 
sion format information. 

11. The audio-video transmitting apparatus according 
to daim 1 . 2, or 3. wherein a default identifier show- 
ing whether to change the contents concerned with 
said transmitting method and/or structure of data to 
be transmitted is added. 

12. The audio-video transmitting apparatus according 
to claim 9. 10, or 1 1 , wherein said identifier or said 
default identifier is added to a predetermined fixed- 
length region of infomnatfon to be transmitted or 
said predetermined position. 

13. An audfo-video receiving apparatus comprising: 



arranging said information without multiplexing it 
before said data and/or control information or trans- 
mit said multiplexing control information through a 
transmission line different from the transmission 
5 line for transmitting said data and/or control infor- 
mation. 

17. An audio-video receiving apparatus comprising: 

10 receiving means for receiving said multiplexing 

control information transmitted from the audio- 
video transmitting apparatus of claim 15 and 
said multiple)^ data and/or control informa- 
tion; and 

15 separating means for separating said multi- 

plexed data and/or control information in 
accordance with said multiplexing control infor- 
mation. 
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receiving means for receiving said transmis- 
sion format information transmitted from the 
audio-video transmitting apparatus of any one 
of claims 1 to 12; and 

transmitted-information interpreting means for 
interpreting said received transnm'sston-format 
information. 

14. The audio-video receiving apparatus according to 
claim 13, wherein 

storing means for storing a plurality of contents 
concerned with said transmitting method 
and/or said structure of data to be transmitted 
and a plurality of its identifiers are included, 
and 

the contents stored in said storing means are 
used to interpret said transmission format infor- 
mation. 

15. An audio-video transmitting apparatus conrprising: 

information multipl&cing means for controlling 
start and end of multiplexing the information for 
a plurality of logical transmission lines for trans- 
mitting data and/or control information is 
included; wherein, 

not only said data and/or control information 
multiplexed by said information multiplexing 
means but also control contents concerned 
with start and end of said multiplexing by said 
information multiplexing means are transmitted 
as multiplexing control information, and 
said data includes video data and/or audio 
data. 

16. The audio-video transmitting apparatus according 
to claim 15, wherein it is possible to select whetiier 
to transmit said multiplexing control information by 



20 1& An audio-video receiving apparatus comprising: 

main iooking-listening means for looking at and 
listening to a broadcast program; and 
auxiliary looking-tistening means for cyclically 
^ detecting tiie state of a broadcast program 

other than tiie broadcast program looked and 
listened through said main Iooking-listening 
means: wherein 

said detection is performed so that a program 
30 and/or data necessary when said broadcast 

program looked and listened through said main 
Iooking-listening means is switched to other 
broadcast program can be snrKX)thly proc- 
essed, and 

3s said data includes video data and/or audio 

data. 

19. The audio-video transmitting apparatus according 
to claim 1, wherein priority values can be changed 

40 in accordance with the situation by transmitting tiie 
offset value of information showing the priority for 
processing of said data. 

20. An audio-video receiving apparatus comprising: 

45 

receiving means for receiving encoded infor- 
mation to which tiie information concerned witii 
the priority for processing under an overload 
state is previously added; and 

50 priority deciding means for deciding a tiireshold 

serving as a criterion for selecting whether to 
process an object in said irrformation received 
by said receiving means; wherein 
the timing for outputting said received informa- 

55 tion is compared with the elapsed time after 

start of processing or the timing for decoding 
said received information is compared with the 
elapsed time after start of processing to 
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change said threshold in accordance with the 
comparison result, and 

video data and/or audio data are or is included 
as said encoding object. 

21. The audio-video receiving apparatus according to 
claim 20, wherein 

retransmission-requestiDriority deciding 
means for deciding a threshold serving as a cri- 
terion Ibr selecting whether to request retrans- 
mission of some of said Information not 
received because it is lost under transmission 
when it is necessary to retransmit said informa- 
tion is included, and 

said decided threshold is decided In accord- 
ance with at least one of the priority controlled 
by said priority deciding means, retransmission 
frequency, lost factor of information, insertion 
interval between in-frame-encoded frames, 
and grading of priority. 

22. An audio-video transmitting apparatus comprising: 

retransmission-priority deciding means for 
deciding a threshold serving as a criterion for 
selecting whether to request retransmission of 
some of said information not received because 
it Is lost under transmission when retransmis- 
sion of said unrecelved information is 
requested is Included, wherein 
said decided threshold Is decided in accord- 
ance with at least one of the priority controlled 
by the priority deciding means of said audio- 
video receiving apparatus of daim 20, retrans- 
mission frequency, lost factor of Information, 
insertion interval between In-frame-encoded 
frames, and grading of priority. 

23. An audio-video transmitting apparatus for transmit- 
ting said encoded information by using the priority 
added to said encoded Information and thereby 
thinning it when (1) an actual transfer rate exceeds 
the target transfer rate of information for a video or 
audio or (2) rt is decided that writing of said 
encoded Information into a transmitting buffer is 
delayed as the result of comparing tiie elapsed time 
after start of transmission with a period to be 
decoded or output added to said encoded informa- 
tion. 

24. A data processing method comprising the steps of: 

inputting a data series including (1) time-series 
data for audio or video, (2) an inter-tlme-series- 
data priority showing the priority of the 
processing between said time-series-data val- 
ues, and (3) a plurality of in-time-series-data 



priorities for dividing said time-series data 
value to show the processing priority between 
divided data values; and 
performing processing by using said inter-time- 
5 series-data priority and said in-time-series- 

data priority togetiier when pluralities of said 
time-series-data values are simultaneously 
present. 

10 25. A data processing apparatus comprising: 



receiving means for receiving a data series 
including (1) time-series data for audio or 
video, (2) an inter-time-series-data priority 
showing the priority of tiie processing between 
said tlme-series-data values, and (3) a plurality 
of in-time-series-data priorities for dividing said 
time-series data value to show the processing 
priority between divided data values; and 
data processing means for performing 
processing by using said inter-time-series-data 
priority and said in-time-series-data priority 
together when pluralities of said time-series- 
data values are simultaneously present. 



IS 
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26. A data processing method comprising the steps of: 

inputting a data series including (1) time-series 
data for audio or video, (2) an inter-time-series- 
data priority showing the priority of the 
processing between said time-series-data val- 
ues, and (3) a plurality of in-time-series-data 
priorities for dividing said time-series data 
value to show the processing priority between 
divided data values; and 
distributing throughput to each of said time- 
series-data values in accordance witii said 
irrter-time-serles<iata priority and moreover, 
adaptively deteriorating tiie processing quality 
of the divided data in said time-series data in 
accordance witii said in-time-serles-data prior- 
ity so that each of said time-series-data values 
Is kept within said distributed throughput. 



45 27. A data processing apparatus comprising: 



receiving means for receiving a data series 
Including (1) time-series data for audio or 
video, (2) an inter-time-series-data priority 
showing the priority of the processing between 
said time-series-data values, and (3) a plurality 
of in-tlme-series-data priorities for dividing said 
time-series data value to show tiie processing 
priority between divided data values; and 
data processing means for distributing through- 
put to each of said time-series-data values in 
accordance with said inter-tlme-series-data pri- 
ority and moreover, adaptively deteriorating the 
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processing quality of the divided data in said 
time-series data in accordance with said in- 
time-series-data priority so that each of said 
time-series-data values is kept within said dis- 
tributed throughput. 5 

28. A data processing method characterized by. when 
an in-time-series-data priority lor a video is added 
every frame of said video and said video for each 
frame is divided into a plurality of packets, io 



29. A data processing apparatus characterized by, 
when an in-time-series-data priority for a video is 
added every frame of said video and said video for 
each frame is divided into a plural'ity of packets. 20 



30. The data processing method according to any one 
of dalms 24, 26, and 28, wherein said in-time- 
series-data priority is described in the header of a 
packet to perform priority processing. 30 

31. The data processing apparatus according to any 
one of claims 25, 27, and 29, wherein said in-time- 
series-data priority is described in the header of a 
packet to perform priority processing. 35 

32. The data processing method according to any one 
of claims 24, 26, and 28, wherein the range of a 
value capable of expressing said in-time-series- 
data priority is made variable to perform prbr'rty 40 
processing. 

33. The data processing apparatus according to any 
one of claims 25, 27, and 29, wherein the range of 

a value capable of expressing said in-time-series- 45 
data priority is made variable to perform priority 
processing. 

34. A data processing method comprising the steps of: 

so 

inputting a data series including time-series 
data for audio or video and an inter-time- 
series-data priority showing the processing pri- 
ority between said time-series data values; and 
processing priorities by using said inter-time- ss 
series-data priority as the value of a relative or 
absolute priority. 



66 

35. A data processing apparatus characterized by: 

inputting a data series including time-series 
data for audio or video and an inter-time- 
series-data priority showing the processing pri- 
ority between said time-series data values; and 
processing priorities by using said inter-time- 
series-data priority as the value of a relative or 
absolute priority. 

36. A data processing method comprising the steps of: 

classifying time-series data values for audio or 
video; 

inputting a data series including said time- 
series data and a plurality of in-time-series- 
data priorities showing the processing priority 
between said classified data values; and 
processing prtorities by using said in-time- 
series-data priority as tiie value of a relative or 
absolute priority. 

37. A data processing apparatus characterized by: 

classifying time-series data values for audio or 
video; 

inputting a data series including said time- 
series data and a plurality of in-time-series- 
data priorities showving the processing priority 
k)etween said classified data values; and 
processing priorities by using said in-time- 
series-data priority as tiie value of a relative or 
absolute priority. 

38. A data processing metiiod comprising the steps of: 

classifying time-series data values for audio or 
video; 

encoding said classified data values; 
inputting a data series describing an in-time- 
series-data priority serving as the value of an 
absolute priority in said encoded information 
and a in-time-series-data priority serving as tiie 
value of a relative priority in tiie header portion 
of a pad^t constituted witii said encoded infor- 
mation; and 
processing priorities. 

39. A data processing apparatus characterized by: 

classifying time-series data values for audio or 
video; 

encoding said classified data values; 
inputting a data series describing an in-time- 
series-data priority serving as tiie value of an 
absolute priority in said encoded information 
and a in-time-series-data priority serving as tiie 
value of a relative priority in the header portion 



EP0 905 976A1 



adding said in-time-series-data priority only to 
tiie header portion of a packet for transmitting 
tiie head portion of a frame of sakl video 
accessible as independent information. is 



adding said in-time-series-data priority only to 
tiie header portion of a packet for transmitting 
tiie head portion of a frame of said video 
accessible as independent information. 25 
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of a packet constrtuted with said encoded infor- 
mation; and 
processing priorities. 

40. A data processing method comprising the steps of: 

inputting a data series including time-series 
data for audio or video and an inter-time- 
series-data priority showing the processing pri- 
ority between time series data values; and 
processing priorities by relating one said inter- 
time-series-data priority or more to a TCP/IP 
logical channel. 

41 . A data processing apparatus characterized by: 

inputting a data series including time-series 
data for audio or video and an inter-time- 
series-data priority showing the processing pri- 
ority between time series data values; and 
processing priorities by relating one said inter- 
time-series-data priority or more to a TCP/IP 
logical channel. 

42. The data processing method according to claim 34 
or 36. wherein 

said priority processing is performed (1) by 
using said inter-time-series-data priority as the 
value of a relative priority when accunfiulating 
and using said inter-time-series-data priority 
and (2) by using said inter-time-series-data pri- 
ority as the value of an absolute priority when 
transmitting said data. 

43. The data processing apparatus according to claim 
35 or 37, wherein 

said priority processing is performed (1) by 
expressing said inter-time-series-data priority 
as the value of a relative priority when accumu- 
lating and using said inter-time-series-data pri- 
ority and (2) by expressing said inter-time- 
series-data priority as the value of an absolute 
priority when transmitting said inter-time- 
series-data priority 

44. The data processing method according to claim 34 
or 36, wherein 

an identifier classifies whether to express the 
value of said priority as a relative value or an 
absolute value. 

45. The data processing apparatus according to claim 
35 or 37, wherein 

an identifier classifies whether to express the 



value of said priority as a relative value or an 
absolute value. 

46. A data processing metiiod comprising the steps of: 

5 

when one time-series data includes a plurality 
of sub-time-series data values, describing the 
relation between said sub-time-series data val- 
ues and thereby defining a metiiod for process- 
10 ing said sub-time-series data to perform priority 

processing. 

47. A data processing apparatus characterized by, 
when one time-series data includes a plurality of 

IS sub-time-series data values, describing tiie relation 
between said sub-time-series data values and 
tiiereby defining a method for processing said sub- 
time-series data to perform priority processing. 

20 48. The data processing method according to any one 
of claims 34, 36, and 46, wherein a packet-consti- 
tuting method is decided in accordance with any 
one of said inter-time-series-data priority, in-time- 
series-data priority, and relational description 
25 between said time-series data values. 

49. The data processing apparatus according to any 
one of claims 35, 37, and 47, wherein a packet-con- 
stituting metiiod is decided in accordance witii any 

30 one of said inter-time-series-data priority, in-time- 
series-data priority, and relational description 
between said time-series data values. 

50. A data processing metiiod characterized by relating 
35 the sliced structure of a video to the structure of a 

packet and tiiereby. making a re-sync marker for 
resynchronization unnecessary 

51. A data processing apparatus characterized by 
40 relating the sliced structure of a video to the struc- 
ture of a packet and tiiereby. making a re-sync 
marker for resynchronization unnecessary. 

52. A data processing apparatus characterized by 
45 transmitting a metiiod for relating time-series data 

for audio or video to a packet together with control 
information or said time-series data and tiiereby, 
defining relating of said time-series data to said 
packet. 

50 

53. The data processing method according to claim 48, 
wherein high en'or protection is applied to a packet 
including the information for said high in-time- 
series-data prior'rty or inter-time-series-data priority 

55 

54. The data processing apparatus according to claim 
49. wherein high error protection is applied to a 
packet including tiie information for said high in- 
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time-series-data priority or inter-time-series-data 
priority. 

55. The data processing method according to claim 34 
or 36, wherein s 



56. The data processing apparatus according to claim 
35 or 37, wherein is 



57. The data processing method according to claim 34 
or 36, wherein 25 



58. The data processing apparatus according to claim 
35 or 37, wherein said priority processing is per- 
formed by assigning a value lower than a character 35 
or control information to said time-series data as 
the value of said in-time-series-data priority or said 
inter-time-series-data priority. 

59. A data processing method comprising tiie steps of: 40 

successively inputting classified time-series 
data and its priority information; and 



60. A data processing apparatus characterized by, suc- 
cessively inputting classified time-series data and ss 
its priority information; and 

(1) when the information for said classified 



time-series data is damaged, performing 
retransmission request processing in order to 
request retransmission of said damaged data 
and, (2) when said classified time-series data 
is continuously or frequentiy lost, applying said 
reti'ansmission request processing only to 
high-priority data. 

61 . A data processing method comprising the step of: 

successively inputting classified time-series 
data and Its priority information; and 
preferentially transmitting said high-priority 
data in accordance with the amount of said 
classified time-series data to be transmitted. 

62. A data processing apparatus characterized by: 

successively inputting classified time-series 
data and its priority infbrmation; and 
preferentially transmitting said high-priority 
data in accordance with the amount of said 
classified time-series data to be transmitted. 

63. A waveform data transmitting method comprising 
tiie steps of: 

(a) dividing a plurality of decoding units consti- 
tuting the waveform-data decoding process 
into a plurality of groups in accordance wrtii tiie 
significance for maintaining quality and count- 
ing the execution frequency of an encoding unit 
corresponding to the decoding unit belonging 
to each group; 

(b) receiving said counted result and transform- 
ing said result into a data string when encoding 
of waveform data for a predetermined period is 
completed; and 

(c) outputting a code which is a waveform-data 
encoding result and said data string and trans- 
mitting the execution frequency of each 
processing unit every a plurality of groups to 
the receiving apparatus. 

64. A waveform data transmitting apparatus compris- 
ing: 

(a) counting means for dividing a plurality of 
decoding units constituting tiie waveform-data 
decoding process into a plurality of groups in 
accordance with the significance for maintain- 
ing quality and counting the execution fre- 
quency of an encoding unit corresponding to 
the decoding unit belonging to each group; 

(b) transforming means for receiving said 
counted result and transforming said result into 
a data string when encoding of waveform data 
for a predetermined period is completed; and 



a priority added to a packet is used as a packet 
priority, and 

said priority processing is performed by relating 
at least eitiier of the values of said In-time- 10 
series-data priority and said inter-time-series- 
data priority to said packet priority. 



a priority added to a packet is used as a packet 
priority, and 

said priority processing is performed by relating 
at least eitiier of the values of said in-time- 20 
series-data priority and said inter-time-series- 
data priority to said packet priority. 



said priority processing is performed by assign- 
ing a value lower tiian a character or control 
information to said time-series data as tiie 
value of said in-time-series<lata priority or said 30 
inter-time-series-data priority. 



(1 ) when the information for said classified 45 
time-series data is damaged, peribrming 
retransmission request processing in order 
to request retransmission of said damaged 
data and (2) when said classified time- 
series data is continuously or frequently so 
lost, applying said retransmission request 
processing only to high-priority data. 
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(c) transmitting means for outputling a code 
which is a wavefbrm<lata encoding result and 
said data string; wherein 
the execution frequency of each processing 
unit is transmitted to the receiving apparatus 
every a plurality of groups. 

65. The waveform data transmitting method according 
to claim 63. wherein 

pluralities of decoding units constituting a plu- 
rality of wavefbrm<iata decoding processes 
are divided into at least one indispensable 
processing or more and at least one dispensa- 
ble processing or more (when this processing 
is omitted, waveforms are deteriorated but 
waveforms can be decoded), the execution fre- 
quency of said indispensable processing and 
that of dispensable processing are counted, 
and the execution frequencies of said indispen- 
sable and dispensable processings for each 
processing unit are transmitted to said receiv- 
ing apparatus. 

66. The waveform data transmitting apparatus accord- 
ing to claim 64. wherein counting means for dividing 
a plurality of decoding units constituting a plurality 
of waveform<lata decoding processes into at least 
one indispensable processing or nnore and at least 
one dispensable processing or more (when this 
processing is omitted, waveforms are deteriorated 
but waveforms can be decoded) and counting the 
execution frequency of said indispensable process- 
ing and that of dispensable processing is included 
and the execution frequencies of said indispensa- 
ble and dispensable processings for each process- 
ing unit are transmitted to said receiving apparatus. 

67. The video waveform data transmitting method 
according to claim 63, wherein a video Is Input as 
said waveform data. 



(a) receiving a data string including the code of 
waveform data and the execution frequency of 
each decoding unit grouped in accordance with 
the significance for maintaining the quality of 
the waveform data decoded from said code 
and outputting said code and said execution 
frequency; 

(b) estimating the execution time of each group 
in accordance with the processing time until 
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68. The video waveform data transmitting apparatus 
according to claim 64, wherein a video is input as 
said waveform data. 45 

69. A waveform data receiving method comprising the 
steps of: 



obtaining a waveform after decoding said code 
and each of said execution frequencies 
obtained from said data string; and 
(c) estimating the processing time required to 
decode a waveform by using the execution fre- 
quency and said execution time, calculating the 
reduced number of execution frequencies of 
groups In which said processing time does not 
exceed the time required to receive said code 
or the time from start of receiving said code up 
to start of receiving the next code (this is 
referred to as designated time) In accordance 
with each execution time output by said receiv- 
ing means and each execution time output by 
said estimating means, estimating the time 
required for decoding, and reducing the execu- 
tion frequency of each group so as to coniplete 
decoding within said designated time. 



20 70. A waveform data receiving apparatus comprising: 



(a) receiving means for receiving a data string 
including the code of waveform data and the 
execution frequency of each decoding unit 
grouped in accordance with the significance for 
maintaining the quality of the waveform data 
decoded from said code and outputting said 
code and said execution frequency; 

(b) estimating means for estimating the execu- 
tion time of each group in accordance with the 
processing time until obtaining a waveform 
after decoding said code and each of said exe- 
cution frequencies obtained from said data 
string; and 

(c) frequency reducing means for estimating 
the processing time required to decode a wave- 
form by using said execution frequency and 
said execution time, calculating the reduced 
number of execution frequencies of the groups 
in which said processing time does not exceed 
the time required to receive said code or the 
time from start of receiving said code up to 
start of receiving the next code (this is referred 
to as designated time) in accordance with each 
execution time output by said receiving means 
and each execution time output by said esti- 
mating means; wherein the time required for 
decoding is estimated and the execution fre- 
quency of each group is reduced so as to com- 
plete decoding within said designated time. 
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71. A waveform data receiving method comprising the 
steps of: 

(a) receiving a data string including the code of 
waveform data and the execution frequencies 
of indispensable and dispensable processings 
for decoding and outputting said code and said 
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execution frequencies; 

(b) estimating the execution frequencies of said 
indispensable and dispensable processings in 
accordance with the processing time until 
obtaining a waveform after decoding said code 
and each of said execution frequencies 
obtained from said data string; and 

(c) estimating the processing time required to 
decode a waveform by using said execution 
frequency and said execution time, calculating 
the reduced number of execution frequencies 
of said dispensable processing in which said 
processing time does not exceed the time 
required to receive said code or the time from 
start of receiving said code up to start of receiv- 
ing the next code (tiiis is referred to as desig- 
nated time) in accordance with each execution 
frequency output by said receiving means and 
each ececution time output by said estimating 
means, estimating tiie time required for decod- 
ing in accordance with each estimated execu- 
tion time, and reducing the execution frequency 
of said dispensable processing so as to com- 
plete decoding within said designated time. 



plete decoding within said designated time. 

73. The video waveform data receiving metiiod accord- 
ing to claim 69, wherein a video is output as said 

5 waveform data. 

74. TTie video waveform data receiving apparatus 
according to claim 70, wherein a video is output as 
said waveform data. 

10 

75. The video waveform data receiving method accord- 
ing to daim 69, wherein (d) the execution time of 
each group obtained through estimation is output 

IS 76. The video waveform data receiving apparatus 
according to daim 70, wherein (d) tiie execution 
time of each group obtained by estinnating means is 
output. 

20 77. The waveform data transmitting method according 
to claim 63, wherein 

(d) a data string including the executfon time of 
each group is input, and 

(e) tiie execution frequency of each group is 
calculated in accordance witii each execution 
time of said receiving means in order to com- 
plete decoding within tiie time required to 
transmit a code lengtii decided by the designa- 
tion i3y a rate controller or the like. 

78. The waveform data transmitting apparatus compris- 
ing: 

(d) receiving means for inputting a data string 
constituted with the ececution time of each 
group; and 

(e) deciding means for calculating the execu- 
tion frequency of each group in accordance 
with each execution time of said receiving 
means in order to complete decoding witiiin the 
time required to transmit a code decided by tiie 
designation by a rate controller or the Wks. 

79. The video waveform data transmitting metiiod 
according to claim 67, wherein 

(d) the execution time of each group Is esti- 
mated in accordance witii the processing time 
required to encode a video and said each exe- 
cution frequency; and 

(e) the processing time required to encode a 
video is estimated by using said execution time 
and the execution frequency of each group is 
calculated in which said processing time does 
not exceed the time usable to process a sheet 
of video determined In accordance with a 
frame rate given as the designation by a user. 



25 

72. A waveform data receiving apparatus comprising: 

(a) receiving means for receiving a data string 
including the code of waveform data and the 
execution frequencies of indispensable and so 
dispensable processings for decoding and out- 
putting said code and said execution frequen- 
cies; 

(b) estimating means for estimating tiie execu- 
tion frequendes of said indispensable and dis- ss 
pensable processings in accordance witii the 
processing time until obtaining a waveform 
after decoding said code and each of said exe- 
cution frequencies obtained from said data 
string; and 40 

(c) frequency reducing means for estimating 
tiie processing time required to decode a wave- 
form by using said execution frequency and 
said execution time and calculating the 
reduced number of execution frequencies of 4S 
said dispensable processing in which said 
processing time does not exceed the time 
required to receive said code or tiie time from 
start of receiving said code up to start of receiv- 
ing the next code (this is referred to as desig- so 
nated time) In accordance with each execution 
frequency output by said receiving means and 
each execution time output by said estimating 
means; wherein 

the time required for decoding is estimated in ss 
accordance with each estimated execution 
time and the execution frequency of said dis- 
pensable processing is reduced so as to conv 
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80. A video waveform data transmitting apparatus 
according to claim 68, wherein 

(d) estimating means for estimating tlie execu- 
tion time of each group in accordance with the 
processing time required to encode a video 
and each execution time output by counting 
means; and 

(e) deciding means for estimating the process- 
ing time required to encode a video by using 
said execution time and calculating the execu- 
tion frequency of each group in which said 
processing time does not exceed the time usa- 
ble to process a sheet of video deterrruned in 
accordance with a frame rate given as the des- 
ignation by a user. 

81. The video waveform data transmitting method 
according to claim 63, wherein said counting result 
and the length of a code corresponding to wave- 
form data for a predetermined period are received 
when generation of said code is completed to trans- 
form the result and length into a data string. 

82. The video waveform data transmitting apparatus 
according to claim 63, wherein transforming means 
is included which receives said counting result of 
said counting means and the length of a code cor- 
responding to waveform data for a predetermined 
period when generation of said code is completed 
to transtonn the result and length into a data string. 

83. The waveform data receiving method according to 
claim 69, wherein a data string including a code 
con^esponding to the waveform data lor a predeter- 
mined period, the execution frequency of each 
decoding unit grouped in accordance with the sig- 
nificance for maintaining the quality of the wave- 
form data decoded from said code, and the length 
of said code is received and said code, execution 
frequency, and code length are output to reduce the 
execution frequency of dispensable processing so 
that the time required for decoding does not exceed 
a code transmission time obtained from the length 
and transmission rate of said code. 

84. The waveform data receiving apparatus according 
to claim 70, wherein receiving means is included 
which receives a data string including a code corre- 
sponding to the waveform data for a predetermined 
period, the execution frequency of each decoding 
unit grouped in accordance with the significance for 
maintaining the quality of the waveform data 
decoded from said code, and the length of said 
code and outputs said code, execution frequency, 
and code length to reduce the execution frequency 
of dispensable processing so that the time required 
for decoding does not exceed a code transmission 



time obtained from the length and transmission rate 
of said code. 

85. Awavelbrm data receiving method for receiving the 
5 code of waveform data and decoding arKi output- 
ting the waveform, comprising the steps of: 

(a) constituting a data string including the des- 
ignation for selecting a processing unit having 
10 an execution time shorter than that of the 

encoding unit included in said code every 
encoding unit corresponding to a processing 
unit constituting the decoding process so that 
the processing time required to decode a wave- 
rs fonri does not exceed the time required to 
receive said code or the time from start of 
receiving said code up to start of receiving the 
next code (this is referred to as designated 
time); and 

20 (b) transmitting said data string to communi- 

cate to the transmitting side that a code for 
completing decoding within said designated 
time is transmitted. 

25 86. A waveform data receiving apparatus for receiving 
the code of waveform data and decoding and out- 
putting said waveform, comprising: 

(a) designated data constituting means for con- 
30 stituting a data string including the designation 

for selecting a processing unit having an execu- 
tion time shorter than that of the encoding unit 
included in said code every encoding unit cor- 
responding to a processing unit constituting the 
35 decoding process so that the processing time 

required to decode a waveform does not 
exceed the time required to receive said code 
or the time from start of receiving said code up 
to start of receiving the next code (this is 
40 refon^ed to as designated time); and 

(b) transmitting means for transmitting said 
data string; wherein 

it Is communicated to the transmitting side that 
a code for corrpleting decoding within said 
45 designated time is transmitted. 

87. A waveform data transmitting method for encoding 
a waveform and oulputting said code, comprising 
the steps of: 

50 

(a) receiving a data string including the desig- 
nation for a processing unit to be selected for 
each processing unit constituting the encoding 
process; and 

55 (b) extracting said designation from said data 

string, encoding a waveform by using the 
processing unit specified in accordance with 
said designation, and outputting a code. 
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88. A waveform data transmitting apparatus for encod- 
ing a waveform and outputting said code, compris- 
ing: 

(a) receiving means for receiving a data string 5 
including tlie designation for a processing unit 

to be selected for eacli processing unit consti- 
tuting the encoding process; and 

(b) extracting means for extracting said desig- 
nation from said data string; wherein io 
a waveform is encoded by using the processing 
unit specified in accordance with said designa- 
tion to output a code. 

89. A waveform data receiving method for receiving the is 
code of waveform data and decoding and output- 
ting a waveform, comprising the steps of: 

(a) counting the execution frequency of each 
processing unit constituting tiie waveform 20 
decoding process; 

(b) estimating the execution time for each 
processing unit in accordance with said execu- 
tion frequency and processing time required to 
decode a waveform; 25 

(c) constituting a data string including tiie des- 
ignation for selecting a processing unit having 
an execution time shorter than that of the 
encoding unit included in said code every 
encoding unit corresponding to the processing 30 
unit constituting tiie decoding process so that 
the processing time required to decode a wave- 
form does not exceed tfie time required to 
receive said code or the time from start of 
receiving said code up to start of receiving the 3S 
next code (tiiis is referred to as designated 
time); and 

(d) transmitting said data string; wherein 

it is communicated to the transmitting mettiod 
tiiat a code for completing decoding witiiin said 40 
designated time is transmitted. 

90. A waveform data receiving apparatus for receiving 
the code of waveform data and decoding and out- 
putting a waveform, comprising: 45 

(a) counting means for counting the execution 
frequency of each processing unit constituting 
tiie waveform decoding process; 

(b) estimating means for estimating tiie execu- so 
tion time for each processing unit in accord- 
ance witii said execution frequency and 
processing time required to decode a wave- 
form; 

(c) designated-data constituting means for con- 55 
stituting a data sfa-ing including tiie designation 

for selecting a processing unit having an execu- 
tion time shorter than tiiat of tiie encoding unit 



included in said code every encoding unit cor- 
responding to the processing unit constituting 
the decoding process so that tiie processing 
time required to decode a waveform does not 
exceed the time required to receive said code 
or the time from start of receiving said code up 
to start of receiving the next code (this is 
referred to as designated time); and 
(d) transmitting means for transmitting said 
data string; wherein 

it is communicated to the transmitting side that 
a code for completing decoding within said 
designated time is transmitted. 

91 . A video waveform data receiving method for receiv- 
ing the code of a video and decoding and outputting 
said video, comprising the steps of: 

(a) constituting a data string including the des- 
ignation for replacing tiie movement compen- 
sating method used to encode a video with tiie 
movement compensation processing having an 
execution time shorter tiian tiiat of the move- 
ment compensation processing included in 
said code so that the processing time required 
to decode a video does not exceed the time 
required to receive said code or the time from 
start of receiving said code up to start of receiv- 
ing tiie next code (this is referred to as desig- 
nated time); and 

(b) transmitting said data string; wherein 

it is communicated to the transmitting side that 
a code for completing encoding within said 
designated time is transmitted. 

92. A video receiving apparatus for receiving the code 
of a video and decoding and outputting said video, 
comprising: 

(a) designated-data constituting means for 
constituting a data string including the designa- 
tion for replacing the movement compensating 
method used to encode a video with the move- 
ment compensation processing having an exe- 
cution time shorter tiian that of the movement 
compensation processing included in said 
code so tiiat tiie processing time required to 
decode a video does not exceed the time 
required to receive said code or the time from 
start of receiving said code up to start of receiv- 
ing the next code (this is refen-ed to as desig- 
nated time); and 

(b) transmitting means for transmitting said 
data string; wherein 

it is communicated to the transmitting side that 
a code for completing encoding within said 
designated time is transmitted. 
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93. A video transmitting method for encoding a video 
and outputting said code, comprising the steps of: 

(a) receiving a data string including the desig- 
nation for the processing to be selected by s 
using the movement compensating processing 
constituting the decoding process; and 

(b) extracting said designation from said data 
string; wherein 

encoding of a video is executed by using the 10 
movement compensating processing specified 
in accordance with said designation to output a 
code. 

94. A video transmitting apparatus for encoding a video is 
and outputting said code, comprising the steps of: 

(a) receiving means for receiving a data string 
including the designation for the processing to 

be selected by using the movement compen- 20 
sating processing constituting the decoding 
process; and 

(b) extracting means for extracting said desig- 
nation from said data string; wherein 

encoding of a video is executed by using the 25 
movement compensating processing specified 
in accordance with said designation to output a 
code. 
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o Information showing start position capable of 
processing data or not 

* Flag for random access (Random access flag), 
e.g. intra-frame (I -picture) In the case of 

picture 

* Flag showing access unit (Access flag), 
e.g« Freme In the case of picture, GOB unit 



AL : Adaptation layer 
ES : Elementary stream 
P T S t Presentation • time • stamp 



Header Data (Picture or sound for each frame) 

Information . 

of data ^ 




• Information showing start position capable of 
processing data or not 

^ • Information showing data reproducing time (PTS) 

• Infozmatlon showing data processing priority 
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F i fif . Ar 

oTSrTransport streamCTransmission packet) 






Information showing start position capable of processing 
pieces of data or not 

: Identification number for showing data sequence(Sequence 
number) 

. • Time concerned with transmission of pieces of data 



©Handling time stamp and marker bit 
(a) 



AL 



Communication 
header 

T f 

Time stamp I 
PTS or not (Additional) 



ES 



(b) 



Communication 
header 

} 



AL 



T 



ES 




Time stamp for 

communication 

header 



Time stamp PTS or not (Additional) 



(c) 



Communication 
header 

T 

Markers it 



AL 



ES 



Substituted by 
AL flag (Additional) 



(d) 



Communication 
header 



AL 



ES 




MarkerBIt of 

communication 

header 



} 

MarkerBit 

(It is interpreted that random access flag 
and access flag are present in AL.) 
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F- ± s , 5(a) 




i « . 5(b) 




I ES I 



AL 



ES 



Communi- 








Communi- 






cation 


AL 


ES 


• • « • 


cation 


AL 


ES 


header 








header 
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I?* i 



H.223 or 
the like 




Data and 

control Infoarmatlon 



UDP/TCP/RTP 




Intra-net and inter-net 

or the like 
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I!* i S . 



Broadcast program transmitting procedure 
<Broadcast type and communication type Including return channel) 
Transmitting side Receiving side 

Transfer of data structure 



ACK/Reject 



(LCN 0) : (*1) 



Transfer of corresponding data 
(From each port) : (*2) 



Are processing and 
reception possible? 
.Start decoding of 
data which can be 
decoded and display 
it. 



<BrGadcast type (with no return channel)) 

Transmitting side Receiving side 

Transfer of program 
information and data structure 
(LCN O) : UDP(*3) 



Transfer of corresponding data 
(From each port) : UDP 



(*1) Must be a system for detecting and retransmitting a paci<et 

loss like TCP. 
(*2) RTP/RTCP or TCP/IP 

(*3) Same data (picture or sound) or control information (broadcast 
program or data structure) is continuously repeatedly 
transmitted. A packet is detected and sequence is kept at a 
receiving terminal in accordance witha sequence number. (To be 
used in a local closed region. Traffic becomes too Large.) 



50 



EP0905976A1 




51 



EP0905 976A1 




52 



EP0905 976A1 



i 



1 O ( a ) 

Receiving terminal 



Control information 
or data 



Program or data 
to be required 



Flag, counter, 
or timer showing 
point of time to be 
required 



Main 

looking-listening - 
section 






Auxi 
Looking- 
sec 


liary 

Listening 

tion 



storing 
section 



Output 

section 



i S . 1 O ( b ) 

Receiving terminal 



Control information 
or data 

> 



IVIain 

looking-listening 
section 



Caption 
broadcast-program 
receiving section 



Output 
section 



Storing 
section 
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i s . 1 X( a ) 



<Hlerarchlcal Image of object> 




<2. Coimmuilcatlon type> 

RTP/RTCP (Program ID of each logical 
channel is fixed. ) 



Terminal 
A 




Terminal 
B 




< — — — — > 


•* ■ — > 

< ■ " » 





LCNO (control) 
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i. S , 1 1( b ) 



-Capability exchange definitlonsCoriglnal from H.245) 



TerminalCapabilitySet 



{ 



sequenceNumber 



::=SEQUENCE 
SequenceNumber, 



multiplexCapability 
capablUtyTable 

capabilityDescriptors 

mpeg4CapabiUty 



MultiplexCapability OPTIONAL 
SET SIZEa.256) OF Capability 
TableEntryOPTIONAL. 
SET SIZE(1..256) OF Capability 
DescriptorOPTIGNAL. 
MPEG4CapabilityOPTIQNAL, 
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-MPEG4 Capability definitions 



MPEG4Capability 



{ 



sequenceNumber 

NumberOfProcessObject 

{ 

MaxNumberOfVideo 
MaxNumberOf Sounds 
MaxNumberOfMux 

} 

reconfigurationALCapability 



} 



IVIPEG4CapabiUtyAck 



{ 



sequenceNumber 



} 

MPEG4CapabiUtyReject 

{ 

sequenceNumber 
NumberOfProcessObject 

maxNumberOfVideo 

maxNumberOf Sounds 

MaxNumberOflViux 

reconf i gurationALCapability 



::=SEQUENCE 

SequenceNumber, 
SEQUENCE 

INTEGER(0..1O23), 
INTEGER(0..1023). 
INTEGER(0..1023). 
BOOLEAN. 

::=SEQUENCE 
SequenceNumber, 

::=SEQUENCE 

SequenceNumber, 
SEQUENCE 

MaxNumberOfVideo, 

l\/taxNumberOf Sounds 
maxNumberOf Mux, 
BOOLEAN. 



1 
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i ss . 13(a) 



—Group MUX definitions 



CreateGroupMux 
{ 

sequenceNumber 
Group MuxID 
lanportNumber 

} 

CreateGroupMuxAck 



i 



} 



sequenceNumber 



CreateGroupMuxReject 
{ 

sequenceNumber 

cause 

{ 



::=SEQUENCE 

SequenceNumber, 

INTEGER(0..1023). 
LANPortNumber. 



::=SEQUENCE 
SequenceNumber, 



::=SEQUENCE 

SequenceNumber. 
CHOICE 



57 



EP0905 976A1 



X s . 1 3( b ) 

DestoryGroupMux 
{ 

sequenceNumber 
GroupMuxID 



DestoryGroupMuxAck 
sequenceNumber 



DestoryGroupMuxReject 
{ 

sequenceNumber 

cause 

{ 

} 

} 



::=SEQUENCE 

SequenceNumber, 
INTEGER(0..1023). 

::=SEQUENCE 
SequenceNumber, 

::=SEQUENCE 

SequenceNumber, 
CHOICE 
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i « 



X 3( c ) 



PortNumberStructure 
I 

sequenceNumber 

lanPortNumber 

numberOfLogicalNumber 

SEQUENCE SIZEa..l5) OF 

* • • 

} 

PortStructureElement 



{ 



LogicalPortNumber 



} 

PortNumberStructureAck 



} 



sequenceNumber 



::=SEQUENCE 

SequenceNumber. 
LANPortNumber. 

rNTEGER(1..l5). 
PortStructureElement, 



::=SEQUENCE 

LogicalPortNumber, 

::=SEQUENCE 
SequenceNumber, 



PortNumberStructureReject 



{ 



sequenceNumber 

cause 

{ 



::=SEQUENCE 

SequenceNumber, 
CHOICE 
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-Logical channel signalling def initions(original from H.245) 
-MPEG4 Object Create Operation(f or LANPortNumber) 



OpenLogicalChannel ::=SEQUENCE 
{ - 

fovwardLogicalChannelNumber LoglcalChannelNumber. 

fowardLogicalChannelParameters SEQUENCE 
{ 

portNumber INTEGER(0..65535)0PTI0NAL. 
dataType DataType. 
multiplexParameters CHOICE 
{ 

h222LogicdLChannelParameters H222LogicalChannelParameters, ' 
h223LogicalChannelParameters H223LogicaiChannelParameters, 
v76LogicaiChannelParameters v76LogicalChannelParanneters, 
' • ' 1 

h2250LogicalChannelParanneters H2250LogicaiaiaTnelParaneters. 
h223AnnexALogicalChannelParameters 
H223AnnexALogicalChannelParameters 
MPEG4Logic*ChannelParameters MPEG4LogicalClianelParameters. 

}, 



t 



} 
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MPEG4LogicalChannelFfeirameters ::=SEQUENCE 



-H.225BASE 
LANportNumber 
ProgramlD 
ProgramName 

} • 

Broadcast ChannelProg ram 



{ 



sequenceNumber 
numberOfChannelNumber 



INTEGER(0..65535), 
INTEGER(0..255). 

0CTETSTR1NG(SIZE(128)), 



::=SEQUENCE 

SequenceNumber, 
INTEGER(0..1023). 



SEQUENCE SIZE(1..1023) OF MPEG4LoglcalChannelParameters 



} 

ChangeLogicalChannelAttribute 



sequenceNumber 

lanportNumber 

ProgramlD 



} 

ChangeLogicalChannelAttributeAck 



i 



1 



sequenceNumber 



{ 



sequenceNumber 

cause 

{ 



::==SEQUENCE 

SequenceNumber 
LANPortNumber. 
INTEGER(0..255)» 



::==SEQUENCE 
SequenceNumber, 



ChangeLogicalChannelAttributeReject ::=SEQUENCE 



SequenceNumber, 
CHOICE 
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i « . X e( a ) 



■MPEG4 Object Class definition 



MPEG4 Object Class definition 



^SEQUENCE 



{ 



] 



sequenceNumber SequenceNumber, 

•ProgramID INTEGER(0.,255). 
NumberOfObjectsList INTEGER(0..1023), 
SEQUENCE SIZE(1..1023) OF ObjectStructureElement 



ObJectStaictLreElennenft 



SSRC ■ 

LANPortNumber 



ScrambleFlag 
CGDOffset 
MediaType 



::=SEQUENCE 

INTEGER(0..16777215). 
INTEGER(1024.5000). 

~forRPT(Video&Sound) 
BOOLEAN. 
INTEGER(0..255)» 
INTEGER(0..255), 



} 



MPEG4 Object Class def initlonAck 



{ 



sequenceNumber 



sequenceNumber 

cause 

{ 



::=SEQUENCE 
SequenceNumber, 



MPEG4 Object Oass definitionReject ::=SEQUENCE 



SequenceNumber, 
CHOICE 
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-Adaptation Layer Reconfiguration Request definitions 



ALReconfiguration 



{ 



sequenceNumber 
RandomAccessFlagMaxBit 
PresentationTimeStampsMaxBIt 
CGDPriorityMaxBIt 



::=CHOICE 

SequenceNumber, 
INTEGER(0...2). . 
INTEGER(0...32). 
INTEGER(0...8), 

— f orVideo and Sound 



-Adaptation Layer Reconfiguration Response definitions 



ALReconfigurationAck 



{ 



sequenceNumber 



} 

ALReconfigurationReject 

sequenceNumber 

cause 

{ 



::=SEQUENCE 
SequenceNumber, 

::=SEQUENCE 

SequenceNumber, 
CHOICE 



<Relatlon between AL, ES, and RTP> 
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i K , IT 



-Setup Program and Data Request definitions 



Setup Request 



{ 



sequenceNumber 

SSRC IMEGER(0..16777215)2^32. 



::=CHOICE 
SequenceNumber, 



Logical Channel Number, 
setupitem 

{ 

executeProgramNumber 
data Number 

exeeuteCommandNumber 
nof itycounter 



flag 

counter 
timer 



INTEGER(1024...5000). 
CHOICE 

INTEGER(0...255). 
INTEGER(0...255), 
INTEGER(0...255). 

CHOICE 

BOOLEAN 

INTEGER(0...255), 
INTEGER(0...255), 
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15* i S . IS 



—control and AL attribute definitions 



ControlALdefinition 
{ 

sequenceNumber 
AL 



{ 



Random AccessFlagUse 

PresentatlonTimeStampUse 

CGDPriorityUse 



::=CHOICE 



SequenceNumber, 
CHOICE 

BOOLEAN, 
BOOLEAN. 
BOOLEAN, 
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i « . 1 9( a ) 

classES_header{ 

uint(4) headerlD; 
uint(24) bufferSizeES; 
uint(l) useTimeStamps; 



} 



u i n t ( 1 6 ) sequenceNumberMaxBit; 
u i nt ( 1 ) useHeaderExtension; 
if (useH.eaderExtenslon){ 



} 

uint(3) reserved: 



uint(l) accessUintStartFlag; 

u i nt ( n randomAccessPointFlag; 

uint(l) OCRsetFlag; 

u i nt (4) degradationPriorityMaxBit; 
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19(b) 



—Adaptation Layer PDU header configuration Request and Command definition 



AL configuration 



{ 



sequenceNumber 
defaultHeaderConfiguration 

headerlD 

MPEG4ALPDUHeaderConfig 



{ 



accessUintStartFlag 

randomAccessPointFlag 

OCRsetFlag 

degradationPriorityMaxBit 



::=SEQUENCE 

SequenceNumber. 
BOOLEAN. 
INTEGER(0..4). 
SEQUENCE 

BOOLEAN. 
BOOLEAN. 
BOOLEAN, 
INTEGER(0..4), 
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I** i S . 2 2 

Processing at receiving terminal under overLoadCCommon to dynamic 
picture and sound) 

Thread for processing sound at system level is previously set it's 
processing priority to a value higiier tlian that of thread for 
processing picture. 

Step 101 
Value of resolution of 
priority to be added to 
* frame, CGDoff set(Can be 
determined in accordance 
with receiving-terminal 
performance) is transmitted 
as control information. 



Frame_skipped 
. =FLASE 



Set priority of 

frame to be 
disused to value 

larger than 
(CutOffPriority) 
"maxPriority*. 



Step 102 



Step 103 




CutOffPriority=0 



CutOffPriority 
=maxPriority 




Step 104 



Frame_sl<ipped=FLASE 



Frame_sl<ipped 
=TRUE 



Deliver data to decoder. 

* 
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i « . 2 a ) 
4031 



.4032 



/ 




/ 




Priority adding 








Priority deciding 






section 








section 











A101.4201 Picture-sound encoder 

4102,4202 Picture-sound decoder 



i « . 2 T( b ) 



Information 
for priority 




Picture frame 
or sound frame 



Communication header 



Payload 
(Divided data) 
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< 
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cn 
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3 X 



RTP header 



H.263 payload header 



H.263 bitstream 



oMode A: GOB, picture boundary 

Presence or absence of mode or PB, start 
and end positions of bit stream, and 
execution timing states of options of 
resolution, frame type, and H.263 



DBQUANT,TR(for B frame), 
TR(f or P frame) 



Core 
'infor- 
mation 



To be set when 

PB frame is present 



oMode B: MB boundary without PB 

Core information for Mode A 

Information for quantization value (GQUANT) , GOB number, 
absolute address of first MB in GOB, and movement vector 
(Horizontal and vertical directions) 

oMode C: MB boundary with PB 

Information for Mode B 

DBQUANT,TR(for B f rame),TR(f or P frame) 



Relating of communication payload 

Mode A + Layer No. +5|^^^Jp. 



Transmitting 
side terminal 



tion 



Configuration information 



Receiving - 
side 

terminal 
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F i « . 3 2 



Data 



Priority at application level 



Dividing 
of data 



Data 



D?ita 



Priority In IP level 



StreamPriority 
FramePriority 



Relating 



Priority of IP 



Priority in data 



Available range 



StreamPriority 


0 - 


- 3 


FramePriority 


0 - 


- 5 


IPV6 


8 - 


- 15 



I CO- • -15] 

^ |*^^^s^ Mapping of part 
- [8- * -15] 



( Lowest ) { Maximum 
priority) 
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EP0905 976A1 



-4. O 

Start 



—1/801 
Input picture. ^ 
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4 1 



Start 

9 

Initialize execution 
time (a_i) of eacli 
element. 



1/ 



901 



Input multiplexed 
data and measure time 
required to input 
multiplexed data. 



'902 



Start of decoding routine 



Output variable-length 
code and data string. 

I 



Fetch execution 
frequency from data 

string and set 
execution frequency 
to x_i. 



903 



.904 



Calculate execution 

frequency in 
accordance with x_i 
and a_i. 



I. 



.905 



Start time 
measurement. 



906 



To decoding 
routine 

1 



VLD 



910 



IDCT 



/ 
^911 




l\/tovement 
compensation 



Complete time 
measurement. 



Estimate execution 
time of each element 
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